pypots.nn package¶

pypots.nn.functional¶

pypots.nn.functional.nonstationary_norm(X, missing_mask=None)[source]¶

Normalization from Non-stationary Transformer. Please refer to [24] for more details.

Parameters:

X (torch.Tensor) – Input data to be normalized. Shape: (n_samples, n_steps (seq_len), n_features).
missing_mask (torch.Tensor, optional) – Missing mask has the same shape as X. 1 indicates observed and 0 indicates missing.

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

X_enc (torch.Tensor) – Normalized data. Shape: (n_samples, n_steps (seq_len), n_features).
means (torch.Tensor) – Means values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).
stdev (torch.Tensor) – Standard deviation values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).

pypots.nn.functional.nonstationary_denorm(X, means, stdev)[source]¶

De-Normalization from Non-stationary Transformer. Please refer to [24] for more details.

Parameters:

X (torch.Tensor) – Input data to be de-normalized. Shape: (n_samples, n_steps (seq_len), n_features).
means (torch.Tensor) – Means values for de-normalization . Shape: (n_samples, n_features) or (n_samples, 1, n_features).
stdev (torch.Tensor) – Standard deviation values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).

Returns:

X_denorm – De-normalized data. Shape: (n_samples, n_steps (seq_len), n_features).

Return type:

torch.Tensor

pypots.nn.functional.calc_mae(predictions, targets, masks=None)[source]¶

Calculate the Mean Absolute Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_mae
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> mae = calc_mae(predictions, targets)

mae = 0.6 here, the error is from the 3rd and 5th elements and is $|3-1|+|5-6|=3$ , so the result is 3/5=0.6.

If we want to prevent some values from MAE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> mae = calc_mae(predictions, targets, masks)

mae = 0.5 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|=1$ , so the result is 1/2=0.5.

pypots.nn.functional.calc_mse(predictions, targets, masks=None)[source]¶

Calculate the Mean Square Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_mse
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> mse = calc_mse(predictions, targets)

mse = 1 here, the error is from the 3rd and 5th elements and is $|3-1|^2+|5-6|^2=5$ , so the result is 5/5=1.

If we want to prevent some values from MSE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> mse = calc_mse(predictions, targets, masks)

mse = 0.5 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|^2=1$ , so the result is 1/2=0.5.

pypots.nn.functional.calc_rmse(predictions, targets, masks=None)[source]¶

Calculate the Root Mean Square Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_rmse
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> rmse = calc_rmse(predictions, targets)

rmse = 1 here, the error is from the 3rd and 5th elements and is $|3-1|^2+|5-6|^2=5$ , so the result is $\sqrt{5/5}=1$ .

If we want to prevent some values from RMSE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> rmse = calc_rmse(predictions, targets, masks)

rmse = 0.707 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|^2=1$ , so the result is $\sqrt{1/2}=0.5$ .

pypots.nn.functional.calc_mre(predictions, targets, masks=None)[source]¶

Calculate the Mean Relative Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_mre
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> mre = calc_mre(predictions, targets)

mre = 0.2 here, the error is from the 3rd and 5th elements and is $|3-1|+|5-6|=3$ , so the result is $\sqrt{3/(1+2+3+4+5)}=1$ .

If we want to prevent some values from MRE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> mre = calc_mre(predictions, targets, masks)

mre = 0.111 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|^2=1$ , so the result is $\sqrt{1/2}=0.5$ .

pypots.nn.functional.calc_quantile_crps(predictions, targets, masks, scaler_mean=0, scaler_stddev=1)[source]¶

Continuous rank probability score for distributional predictions.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor]) – The masks for filtering the specific values in inputs and target from evaluation. Only values at corresponding positions where values ==1 in masks will be used for evaluation.
scaler_mean – Mean value of the scaler used to scale the data.
scaler_stddev – Standard deviation value of the scaler used to scale the data.

Returns:

Value of continuous rank probability score.

Return type:

CRPS

pypots.nn.functional.calc_quantile_crps_sum(predictions, targets, masks, scaler_mean=0, scaler_stddev=1)[source]¶

Sum continuous rank probability score for distributional predictions.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor]) – The masks for filtering the specific values in inputs and target from evaluation. Only values at corresponding positions where values ==1 in masks will be used for evaluation.
scaler_mean – Mean value of the scaler used to scale the data.
scaler_stddev – Standard deviation value of the scaler used to scale the data.

Returns:

Sum value of continuous rank probability score.

Return type:

CRPS

pypots.nn.functional.calc_binary_classification_metrics(prob_predictions, targets, pos_label=1)[source]¶

Calculate the evaluation metrics for the binary classification task, including accuracy, precision, recall, f1 score, area under ROC curve, and area under Precision-Recall curve. If targets contains multiple categories, please set the positive category as pos_label.

Parameters:

prob_predictions (ndarray) – Estimated probability predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int) – The label of the positive class. Note that pos_label is also the index used to extract binary prediction probabilities from predictions.

Returns:

A dictionary contains classification metrics and useful results:

predictions: binary categories of the prediction results;

accuracy: prediction accuracy;

precision: prediction precision;

recall: prediction recall;

f1: F1-score;

precisions: precision values of Precision-Recall curve

recalls: recall values of Precision-Recall curve

pr_auc: area under Precision-Recall curve

fprs: false positive rates of ROC curve

tprs: true positive rates of ROC curve

roc_auc: area under ROC curve

Return type:

classification_metrics

pypots.nn.functional.calc_precision_recall_f1(prob_predictions, targets, pos_label=1)[source]¶

Calculate precision, recall, and F1-score of model predictions.

Parameters:

prob_predictions (ndarray) – Estimated probability predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int, default=1) – The label of the positive class.

Return type:

Tuple[float, float, float]

Returns:

precision – The precision value of model predictions.
recall – The recall value of model predictions.
f1 – The F1 score of model predictions.

pypots.nn.functional.calc_pr_auc(prob_predictions, targets, pos_label=1)[source]¶

Calculate precisions, recalls, and area under PR curve of model predictions.

Parameters:

prob_predictions (ndarray) – Estimated probability predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int, default=1) – The label of the positive class.

Return type:

Tuple[float, ndarray, ndarray, ndarray]

Returns:

pr_auc – Value of area under Precision-Recall curve.
precisions – Precision values of Precision-Recall curve.
recalls – Recall values of Precision-Recall curve.
thresholds – Increasing thresholds on the decision function used to compute precision and recall.

pypots.nn.functional.calc_roc_auc(prob_predictions, targets, pos_label=1)[source]¶

Calculate false positive rates, true positive rates, and area under AUC curve of model predictions.

Parameters:

prob_predictions (ndarray) – Estimated probabilities/predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int, default=1) – The label of the positive class.

Return type:

Tuple[float, ndarray, ndarray, ndarray]

Returns:

roc_auc – The area under ROC curve.
fprs – False positive rates of ROC curve.
tprs – True positive rates of ROC curve.
thresholds – Increasing thresholds on the decision function used to compute FPR and TPR.

pypots.nn.functional.calc_acc(class_predictions, targets)[source]¶

Calculate accuracy score of model predictions.

Parameters:

class_predictions (ndarray) – Estimated classification predictions returned by a classifier.
targets (ndarray) – Ground truth (correct) classification results.

Returns:

The accuracy of model predictions.

Return type:

acc_score

pypots.nn.functional.calc_rand_index(class_predictions, targets)[source]¶

Calculate Rand Index, a measure of the similarity between two data clusterings. Refer to [47].

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

Rand index.

Return type:

References

pypots.nn.functional.calc_adjusted_rand_index(class_predictions, targets)[source]¶

Calculate adjusted Rand Index.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

Adjusted Rand index.

Return type:

aRI

References

pypots.nn.functional.calc_cluster_purity(class_predictions, targets)[source]¶

Calculate cluster purity.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

cluster purity.

Return type:

cluster_purity

Notes

This function is from the answer https://stackoverflow.com/a/51672699 on StackOverflow.

pypots.nn.functional.calc_nmi(class_predictions, targets)[source]¶

Calculate Normalized Mutual Information between two clusterings.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

NMI – Normalized Mutual Information

Return type:

float,

pypots.nn.functional.calc_chs(X, predicted_labels)[source]¶

Compute the Calinski and Harabasz score (also known as the Variance Ratio Criterion).

Xarray-like of shape (n_samples_a, n_features): A feature array, or learned latent representation, that can be used for clustering.
predicted_labelsarray-like of shape (n_samples): Predicted labels for each sample.

Returns:: calinski_harabasz_score – The resulting Calinski-Harabasz score. In short, the higher, the better.
Return type:: float

References

pypots.nn.functional.calc_dbs(X, predicted_labels)[source]¶

Compute the Davies-Bouldin score.

Parameters:

X (array-like of shape (n_samples_a, n_features)) – A feature array, or learned latent representation, that can be used for clustering.
predicted_labels (array-like of shape (n_samples)) – Predicted labels for each sample.

Returns:

davies_bouldin_score – The resulting Davies-Bouldin score. In short, the lower, the better.

Return type:

float

References

pypots.nn.functional.calc_silhouette(X, predicted_labels)[source]¶

Compute the mean Silhouette Coefficient of all samples.

Parameters:

X (array-like of shape (n_samples_a, n_features)) – A feature array, or learned latent representation, that can be used for clustering.
predicted_labels (array-like of shape (n_samples)) – Predicted labels for each sample.

Returns:

silhouette_score – Mean Silhouette Coefficient for all samples. In short, the higher, the better.

Return type:

float

References

pypots.nn.functional.calc_internal_cluster_validation_metrics(X, predicted_labels)[source]¶

Computer all internal cluster validation metrics available in PyPOTS and return as a dictionary.

Parameters:

X (array-like of shape (n_samples_a, n_features)) – A feature array, or learned latent representation, that can be used for clustering.
predicted_labels (array-like of shape (n_samples)) – Predicted labels for each sample.

Returns:

internal_cluster_validation_metrics – A dictionary contains all internal cluster validation metrics available in PyPOTS.

Return type:

dict

pypots.nn.functional.calc_external_cluster_validation_metrics(class_predictions, targets)[source]¶

Computer all external cluster validation metrics available in PyPOTS and return as a dictionary.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

external_cluster_validation_metrics – A dictionary contains all external cluster validation metrics available in PyPOTS.

Return type:

dict

pypots.nn.modules.autoformer¶

The package including the modules of Autoformer.

Refer to the paper Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting. In Advances in Neural Information Processing Systems, volume 34, pages 22419–22430. Curran Associates, Inc., 2021.

Notes

This implementation is inspired by the official one https://github.com/thuml/Autoformer

class pypots.nn.modules.autoformer.AutoCorrelation(factor=1, attention_dropout=0.1)[source]¶

AutoCorrelation Mechanism with the following two phases:

period-based dependencies discovery
time delay aggregation

This block can replace the self-attention family mechanism seamlessly.

time_delay_agg_training(values, corr)[source]¶: SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the training phase.

time_delay_agg_inference(values, corr)[source]¶: SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the inference phase.

time_delay_agg_full(values, corr)[source]¶: Standard version of Autocorrelation

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class pypots.nn.modules.autoformer.SeasonalLayerNorm(n_channels)[source]¶

A special designed layer normalization for the seasonal part.

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.MovingAvgBlock(kernel_size, stride)[source]¶

The moving average block to highlight the trend of time series.

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.SeriesDecompositionBlock(kernel_size)[source]¶

Series decomposition block

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.AutoformerEncoderLayer(attn_opt, d_model, n_heads, d_ffn, moving_avg=25, dropout=0.1, activation='relu')[source]¶

Autoformer encoder layer with the progressive decomposition architecture.

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.AutoformerDecoderLayer(self_attn_opt, cross_attn_opt, d_model, n_heads, d_out, d_ff=None, moving_avg=25, dropout=0.1, activation='relu')[source]¶

Autoformer decoder layer with the progressive decomposition architecture

forward(x, cross, x_mask=None, cross_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.AutoformerEncoder(n_layers, d_model, n_heads, d_ffn, factor, moving_avg_window_size, dropout, activation='relu')[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.brits¶

The package including the modules of BRITS.

Refer to the paper Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. BRITS: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

Notes

This implementation is inspired by the official one https://github.com/caow13/BRITS The bugs in the original implementation are fixed here.

class pypots.nn.modules.brits.BackboneRITS(n_steps, n_features, rnn_hidden_size, training_loss=MAE())[source]¶

model RITS: Recurrent Imputation for Time Series

Attributes:

n_steps – sequence length (number of time steps)
n_features – number of features (input dimensions)
rnn_hidden_size – the hidden size of the RNN cell
rnn_cell – the LSTM cell to model temporal data
temp_decay_h – the temporal decay module to decay RNN hidden state
temp_decay_x – the temporal decay module to decay data in the raw feature space
hist_reg – the temporal-regression module to project RNN hidden state into the raw feature space
feat_reg – the feature-regression module
combining_weight – the module used to generate the weight to combine history regression and feature regression

Parameters:

n_steps (int) – sequence length (number of time steps)
n_features (int) – number of features (input dimensions)
rnn_hidden_size (int) – the hidden size of the RNN cell

forward(inputs, direction)[source]¶

Parameters:

inputs (dict) – Input data, a dictionary includes feature values, missing masks, and time-gap values.
direction (str) – A keyword to extract data from inputs.

Return type:

Tuple[Tensor, Tensor, Tensor, Tensor]

Returns:

imputed_data – Input data with missing parts imputed. Shape of [batch size, sequence length, feature number].
estimations – Reconstructed data. Shape of [batch size, sequence length, feature number].
hidden_states (tensor,) – [batch size, RNN hidden size]
reconstruction_loss – reconstruction loss

class pypots.nn.modules.brits.BackboneBRITS(n_steps, n_features, rnn_hidden_size, training_loss=MAE())[source]¶

model BRITS: Bidirectional RITS BRITS consists of two RITS, which take time-series data from two directions (forward/backward) respectively.

Attributes:

n_steps – sequence length (number of time steps)
n_features – number of features (input dimensions)
rnn_hidden_size – the hidden size of the RNN cell
rits_f (RITS object) – the forward RITS model
rits_b (RITS object) – the backward RITS model

forward(inputs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

class pypots.nn.modules.brits.FeatureRegression(input_size)[source]¶

The module used to capture the correlation between features for imputation in BRITS.

Attributes:

W (tensor) – The weights (parameters) of the module.
b (tensor) – The bias of the module.
m (buffer) (tensor) – The mask matrix, a squire matrix with diagonal entries all zeroes while left parts all ones. It is applied to the weight matrix to mask out the estimation contributions from features themselves. It is used to help enhance the imputation performance of the network.

Parameters:

input_size (the feature dimension of the input)

forward(x)[source]¶

Forward processing of the NN module.

Parameters:: x (tensor,) – the input for processing
Returns:: output – the processed result containing imputation from feature regression
Return type:: tensor,

pypots.nn.modules.crli¶

The package including the modules of CRLI.

Refer to the paper Qianli Ma, Chuxin Chen, Sen Li, and Garrison W. Cottrell. Learning Representations for Incomplete Time Series Clustering. In AAAI, 35(10):8837–8846, May 2021.

Notes

This implementation is inspired by the official one https://github.com/qianlima-lab/CRLI

class pypots.nn.modules.crli.BackboneCRLI(n_steps, n_features, n_generator_layers, rnn_hidden_size, decoder_fcn_output_dims, rnn_cell_type='GRU')[source]¶

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

class pypots.nn.modules.crli.CrliGenerator(n_layers, n_features, d_hidden, cell_type)[source]¶

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.crli.CrliDecoder(n_steps, d_input, d_output, fcn_output_dims=None)[source]¶

forward(generator_fb_hidden_states)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.crli.CrliDiscriminator(cell_type, d_input)[source]¶

forward(X, missing_mask, imputation_latent)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

pypots.nn.modules.crossformer¶

The package including the modules of Crossformer.

Refer to the paper Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The 11th ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/Thinklab-SJTU/Crossformer

class pypots.nn.modules.crossformer.CrossformerEncoder(attn_layers)[source]¶

forward(x, src_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.CrossformerDecoder(layers)[source]¶

forward(x, cross)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.TwoStageAttentionLayer(seg_num, factor, d_model, n_heads, d_k, d_v, d_ff=None, dropout=0.1, attn_dropout=0.1)[source]¶

The Two Stage Attention (TSA) Layer input/output shape: [batch_size, Data_dim(D), Seg_num(L), d_model]

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.ScaleBlock(win_size, d_model, n_heads, d_ff, depth, dropout, seg_num, factor)[source]¶

forward(x, attn_mask=None, tau=None, delta=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.CrossformerDecoderLayer(self_attention, cross_attention, seg_len, d_model, d_ff=None, dropout=0.1)[source]¶

forward(x, cross)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.csdi¶

The package including the modules of CSDI.

Refer to the paper Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. In NeurIPS, 2021.

Notes

This implementation is inspired by the official one the official implementation https://github.com/ermongroup/CSDI.

class pypots.nn.modules.csdi.BackboneCSDI(n_layers, n_heads, n_channels, d_target, d_time_embedding, d_feature_embedding, d_diffusion_embedding, is_unconditional, n_diffusion_steps, schedule, beta_start, beta_end)[source]¶

forward(observed_data, cond_mask, side_info, n_sampling_times)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csdi.CsdiDiffusionEmbedding(n_diffusion_steps, d_embedding=128, d_projection=None)[source]¶

forward(diffusion_step)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csdi.CsdiDiffusionModel(n_diffusion_steps, d_diffusion_embedding, d_input, d_side, n_channels, n_heads, n_layers)[source]¶

forward(x, cond_info, diffusion_step)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csdi.CsdiResidualBlock(d_side, n_channels, diffusion_embedding_dim, nheads)[source]¶

forward(x, cond_info, diffusion_emb)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.dlinear¶

The package including the modules of DLinear.

Refer to the paper Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, volume 37, pages 11121–11128, Jun. 2023.

Notes

This implementation is inspired by the official one https://github.com/cure-lab/LTSF-Linear

class pypots.nn.modules.dlinear.BackboneDLinear(n_steps, n_features, individual=False, d_model=None)[source]¶

forward(seasonal_init, trend_init)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.etsformer¶

The package including the modules of ETSformer.

Refer to the paper Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. ETSformer: Exponential smoothing transformers for time-series forecasting. In ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/salesforce/ETSformer

class pypots.nn.modules.etsformer.ETSformerEncoder(layers)[source]¶

forward(res, level, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.etsformer.ETSformerEncoderLayer(d_model, n_heads, d_out, seq_len, pred_len, k, d_ffn=None, dropout=0.1, activation='sigmoid', layer_norm_eps=1e-05)[source]¶

forward(res, level, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.etsformer.ETSformerDecoder(layers)[source]¶

forward(growths, seasons)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.etsformer.ETSformerDecoderLayer(d_model, n_heads, d_out, pred_len, dropout=0.1)[source]¶

forward(growth, season)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.fedformer¶

The package including the modules of FEDformer.

Refer to the paper Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 27268–27286. PMLR, 17–23 Jul 2022.

Notes

This implementation is inspired by the official one https://github.com/MAZiqing/FEDformer

class pypots.nn.modules.fedformer.FEDformerEncoder(n_steps, n_layers, d_model, n_heads, d_ffn, moving_avg_window_size, dropout, version='Fourier', modes=32, mode_select='random', activation='relu')[source]¶

forward(X, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.fedformer.FEDformerDecoder(n_steps, n_pred_steps, n_layers, n_heads, d_model, d_ffn, d_output, moving_avg_window_size, dropout, version='Fourier', modes=32, mode_select='random', activation='relu')[source]¶

forward(X, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.fedformer.MultiWaveletTransform(ich=1, k=8, alpha=16, c=128, nCZ=1, L=0, base='legendre', attention_dropout=0.1)[source]¶

1D multiwavelet block.

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

class pypots.nn.modules.fedformer.MultiWaveletCross(in_channels, out_channels, seq_len_q, seq_len_kv, modes, c=64, k=8, ich=512, L=0, base='legendre', mode_select_method='random', initializer=None, activation='tanh', **kwargs)[source]¶

1D Multiwavelet Cross Attention layer.

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

class pypots.nn.modules.fedformer.FourierBlock(in_channels, out_channels, seq_len, modes=0, mode_select_method='random')[source]¶

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

class pypots.nn.modules.fedformer.FourierCrossAttention(in_channels, out_channels, seq_len_q, seq_len_kv, modes=64, mode_select_method='random', activation='tanh', policy=0, num_heads=8)[source]¶

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

pypots.nn.modules.gpvae¶

The package including the modules of GP-VAE.

Refer to the paper Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pages 1651–1661. PMLR, 2020.

Notes

This implementation is inspired by the official one https://github.com/ratschlab/GP-VAE

class pypots.nn.modules.gpvae.BackboneGPVAE(input_dim, time_length, latent_dim, encoder_sizes=(64, 64), decoder_sizes=(64, 64), beta=1, M=1, K=1, kernel='cauchy', sigma=1.0, length_scale=7.0, kernel_scales=1, window_size=24)[source]¶

model GPVAE with Gaussian Process prior

Parameters:

input_dim (int,) – the feature dimension of the input
time_length (int,) – the length of each time series
latent_dim (int,) – the feature dimension of the latent embedding
encoder_sizes (tuple,) – the tuple of the network size in encoder
decoder_sizes (tuple,) – the tuple of the network size in decoder
beta (float,) – the weight of the KL divergence
M (int,) – the number of Monte Carlo samples for ELBO estimation
K (int,) – the number of importance weights for IWAE model
kernel (str,) – the Gaussian Process kernel [“cauchy”, “diffusion”, “rbf”, “matern”]
sigma (float,) – the scale parameter for a kernel function
length_scale (float,) – the length scale parameter for a kernel function
kernel_scales (int,) – the number of different length scales over latent space dimensions

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.grud¶

The package including the modules of GRU-D.

Refer to the paper Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports, 8(1):6085, April 2018.

Notes

This implementation is inspired by the official one https://github.com/PeterChe1990/GRU-D

class pypots.nn.modules.grud.BackboneGRUD(n_steps, n_features, rnn_hidden_size)[source]¶

forward(X, missing_mask, deltas, empirical_mean, X_filledLOCF)[source]¶

Forward processing of GRU-D.

Parameters:

X
missing_mask
deltas
empirical_mean
X_filledLOCF

Return type:

Tuple[Tensor, ...]

Returns:

classification_pred
logits

class pypots.nn.modules.grud.TemporalDecay(input_size, output_size, diag=False)[source]¶

The module used to generate the temporal decay factor gamma in the GRU-D model. Please refer to the original paper [42] for more details.

Attributes:

W (tensor,) – The weights (parameters) of the module.
b (tensor,) – The bias of the module.

Parameters:

input_size (int,) – the feature dimension of the input
output_size (int,) – the feature dimension of the output
diag (bool,) – whether to product the weight with an identity matrix before forward processing

References

forward(delta)[source]¶

Forward processing of this NN module.

Parameters:: delta (tensor, shape [n_samples, n_steps, n_features]) – The time gaps.
Returns:: gamma – The temporal decay factor.
Return type:: tensor, of the same shape with parameter delta, values in (0,1]

pypots.nn.modules.informer¶

The package including the modules of Informer.

Refer to the paper Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115, 2021.

Notes

This implementation is inspired by the official one https://github.com/zhouhaoyi/Informer2020

class pypots.nn.modules.informer.ProbAttention(mask_flag=True, factor=5, attention_dropout=0.1, scale=None)[source]¶

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.ConvLayer(c_in)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerEncoderLayer(attention, d_model, d_ff=None, dropout=0.1, activation='relu')[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerDecoderLayer(self_attention, cross_attention, d_model, d_ff=None, dropout=0.1, activation='relu')[source]¶

forward(x, cross, x_mask=None, cross_mask=None, tau=None, delta=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerEncoder(attn_layers, conv_layers=None, norm_layer=None)[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerDecoder(layers, norm_layer=None, projection=None)[source]¶

forward(x, cross, x_mask=None, cross_mask=None, trend=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.mrnn¶

The package including the modules of M-RNN.

Refer to the paper Jinsung Yoon, William R. Zame, and Mihaela van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 66(5):14771490, 2019.

Notes

This implementation is inspired by the official one https://github.com/jsyoon0823/MRNN and https://github.com/WenjieDu/SAITS

class pypots.nn.modules.mrnn.BackboneMRNN(n_steps, n_features, rnn_hidden_size)[source]¶

forward(inputs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor]

Note

class pypots.nn.modules.mrnn.MrnnFcnRegression(feature_num)[source]¶

M-RNN fully connection regression Layer

forward(x, missing_mask, target)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.patchtst¶

The package including the modules of PatchTST.

Refer to the paper Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/yuqinie98/PatchTST

class pypots.nn.modules.patchtst.PatchtstEncoder(n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.patchtst.PatchEmbedding(d_model, patch_size, patch_stride, padding, dropout, positional_embedding=True)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.patchtst.RegressionHead(n_features, d_model, d_output, head_dropout, y_range=None)[source]¶

forward(x)[source]¶: x: [bs x nvars x d_model x num_patch] output: [bs x output_dim]

class pypots.nn.modules.patchtst.ClassificationHead(n_features, d_model, n_classes, head_dropout)[source]¶

forward(x)[source]¶: x: [bs x nvars x d_model x num_patch] output: [bs x n_classes]

class pypots.nn.modules.patchtst.PredictionHead(d_model, n_patches, n_steps_forecast, head_dropout=0, individual=False, n_features=0)[source]¶

forward(x)[source]¶: x: [bs x nvars x d_model x num_patch] output: [bs x forecast_len x nvars]

pypots.nn.modules.raindrop¶

The package including the modules of Raindrop.

Refer to the paper Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In ICLR, 2022.

Notes

This implementation is inspired by the official one the official implementation https://github.com/mims-harvard/Raindrop

class pypots.nn.modules.raindrop.BackboneRaindrop(n_features, n_layers, d_model, n_heads, d_ffn, n_classes, dropout=0.3, max_len=215, d_static=9, d_pe=16, aggregation='mean', sensor_wise_mask=False, static=False)[source]¶

forward(X, timestamps, lengths)[source]¶

Forward processing of Raindrop.

Parameters:

X (Tensor) – The input tensor of shape (batch_size, n_features, max_len).
timestamps (Tensor) – The timestamps tensor of shape (batch_size, max_len).
lengths (Tensor) – The lengths tensor of shape (batch_size, 1).

Returns:

prediction

Return type:

torch.Tensor

pypots.nn.modules.saits¶

The package including the modules of SAITS.

Refer to the paper Wenjie Du, David Cote, and Yan Liu. SAITS: Self-Attention-based Imputation for Time Series. Expert Systems with Applications, 219:119619, 2023.

Notes

This implementation is inspired by the official one https://github.com/WenjieDu/SAITS

class pypots.nn.modules.saits.BackboneSAITS(n_steps, n_features, n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

forward(X, missing_mask, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.saits.SaitsEmbedding(d_in, d_out, with_pos, n_max_steps=1000, dropout=0)[source]¶

The embedding method from the SAITS paper [1].

Parameters:

d_in (int) – The input dimension.
d_out (int) – The output dimension.
with_pos (bool) – Whether to add positional encoding.
n_max_steps (int) – The maximum number of steps. It only works when with_pos is True.
dropout (float) – The dropout rate.

forward(X, missing_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.saits.SaitsLoss(ORT_weight, MIT_weight, loss_calc_func=MAE())[source]¶

forward(reconstruction, X_ori, missing_mask, indicating_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.timesnet¶

The package including the modules of TimesNet.

Refer to the paper Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/thuml/Time-Series-Library

class pypots.nn.modules.timesnet.BackboneTimesNet(n_layers, n_steps, n_pred_steps, top_k, d_model, d_ffn, n_kernels)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

class pypots.nn.modules.timesnet.InceptionBlockV1(in_channels, out_channels, num_kernels=6, init_weight=True)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.timesnet.TimesBlock(seq_len, pred_len, top_k, d_model, d_ffn, num_kernels)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.transformer¶

The package including the modules of Transformer.

Refer to the papers Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. and Wenjie Du, David Cote, and Yan Liu. SAITS: Self-Attention-based Imputation for Time Series. Expert Systems with Applications, 219:119619, 2023.

Notes

This implementation is inspired by https://github.com/WenjieDu/SAITS

class pypots.nn.modules.transformer.ScaledDotProductAttention(temperature, attn_dropout=0.1)[source]¶

Scaled dot-product attention.

Parameters:

temperature (float) – The temperature for scaling.
attn_dropout (float) – The dropout rate for the attention map.

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Forward processing of the scaled dot-product attention.

Parameters:

q (Tensor) – Query tensor.
k (Tensor) – Key tensor.
v (Tensor) – Value tensor.
attn_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps]. 0 in attn_mask means values at the according position in the attention map will be masked out.

Return type:

Tuple[Tensor, Tensor]

Returns:

output – The result of Value multiplied with the scaled dot-product attention map.
attn – The scaled dot-product attention map.

class pypots.nn.modules.transformer.MultiHeadAttention(attn_opt, d_model, n_heads, d_k, d_v)[source]¶

Transformer multi-head attention module.

Parameters:

attn_opt (AttentionOperator) – The attention operator, e.g. the self-attention proposed in Transformer.
d_model (int) – The dimension of the input tensor.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.

forward(q, k, v, attn_mask, **kwargs)[source]¶

Forward processing of the multi-head attention module.

Parameters:

q (Tensor) – Query tensor.
k (Tensor) – Key tensor.
v (Tensor) – Value tensor.
attn_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps]. 0 in attn_mask means values at the according position in the attention map will be masked out.

Return type:

Tuple[Tensor, Tensor]

Returns:

v – The output of the multi-head attention layer.
attn_weights – The attention map.

class pypots.nn.modules.transformer.PositionalEncoding(d_hid, n_positions=1000)[source]¶

The original positional-encoding module for Transformer.

Parameters:

d_hid (int) – The dimension of the hidden layer.
n_positions (int) – The max number of positions.

forward(x, dim=1, return_only_pos=False)[source]¶

Forward processing of the positional encoding module.

Parameters:

x (Tensor) – Input tensor.
dim (int) – The dimension to add the positional encoding.
return_only_pos (bool) – Whether to return only the positional encoding.

Return type:

Tensor

Returns:

If return_only_pos is True –

pos_enc:
The positional encoding.
else –

x_with_pos:
Output tensor, the input tensor with the positional encoding added.

class pypots.nn.modules.transformer.TransformerEncoderLayer(attn_opt, d_model, n_heads, d_k, d_v, d_ffn, dropout=0.1)[source]¶

Transformer encoder layer.

Parameters:

attn_opt (AttentionOperator) – The attention operator for the multi-head attention module in the encoder layer.
d_model (int) – The dimension of the input tensor.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer.
dropout (float) – The dropout rate.

forward(enc_input, src_mask=None, **kwargs)[source]¶

Forward processing of the encoder layer.

Parameters:

enc_input (Tensor) – Input tensor.
src_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Tuple[Tensor, Tensor]

Returns:

enc_output – Output tensor.
attn_weights – The attention map.

class pypots.nn.modules.transformer.TransformerDecoderLayer(slf_attn_opt, enc_attn_opt, d_model, n_heads, d_k, d_v, d_ffn, dropout=0.1)[source]¶

Transformer decoder layer.

Parameters:

slf_attn_opt (AttentionOperator) – The attention operator for the multi-head attention module in the decoder layer.
enc_attn_opt (AttentionOperator) – The attention operator for the encoding multi-head attention module in the decoder layer.
d_model (int) – The dimension of the input tensor.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer.
dropout (float) – The dropout rate.

forward(dec_input, enc_output, slf_attn_mask=None, dec_enc_attn_mask=None, **kwargs)[source]¶

Forward processing of the decoder layer.

Parameters:

dec_input (Tensor) – Input tensor.
enc_output (Tensor) – Output tensor from the encoder.
slf_attn_mask (Optional[Tensor]) – Masking tensor for the self-attention module. The shape should be [batch_size, n_heads, n_steps, n_steps].
dec_enc_attn_mask (Optional[Tensor]) – Masking tensor for the encoding attention module. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

dec_output – Output tensor.
dec_slf_attn – The self-attention map.
dec_enc_attn – The encoding attention map.

class pypots.nn.modules.transformer.PositionWiseFeedForward(d_in, d_hid, dropout=0.1)[source]¶

Position-wise feed forward network (FFN) in Transformer.

Parameters:

d_in (int) – The dimension of the input tensor.
d_hid (int) – The dimension of the hidden layer.
dropout (float) – The dropout rate.

forward(x)[source]¶

Forward processing of the position-wise feed forward network.

Parameters:: x (Tensor) – Input tensor.
Returns:: Output tensor.
Return type:: x

class pypots.nn.modules.transformer.TransformerEncoder(n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

Transformer encoder.

Parameters:

n_layers (int) – The number of layers in the encoder.
d_model (int) – The dimension of the module manipulation space. The input tensor will be projected to a space with d_model dimensions.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer in the feed-forward network.
dropout (float) – The dropout rate.
attn_dropout (float) – The dropout rate for the attention map.

forward(x, src_mask=None)[source]¶

Forward processing of the encoder.

Parameters:

x (Tensor) – Input tensor.
src_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Union[Tensor, Tuple[Tensor, list]]

Returns:

enc_output – Output tensor.
attn_weights_collector – A list containing the attention map from each encoder layer.

class pypots.nn.modules.transformer.TransformerDecoder(n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

Transformer decoder.

Parameters:

n_layers (int) – The number of layers in the decoder.
d_model (int) – The dimension of the module manipulation space. The input tensor will be projected to a space with d_model dimensions.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer in the feed-forward network.
dropout (float) – The dropout rate.
attn_dropout (float) – The dropout rate for the attention map.

forward(trg_seq, enc_output, trg_mask=None, src_mask=None)[source]¶

Forward processing of the decoder.

Parameters:

trg_seq (Tensor) – Input tensor.
enc_output (Tensor) – Output tensor from the encoder.
trg_mask (Optional[Tensor]) – Masking tensor for the self-attention module.
src_mask (Optional[Tensor]) – Masking tensor for the encoding attention module.

Return type:

Union[Tensor, Tuple[Tensor, list, list]]

Returns:

dec_output – Output tensor.
dec_slf_attn_collector – A list containing the self-attention map from each decoder layer.
dec_enc_attn_collector – A list containing the encoding attention map from each decoder layer.

pypots.nn.modules.usgan¶

The package including the modules of USGAN.

Refer to the paper Xiaoye Miao, Yangyang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative Semi-supervised Learning for Multivariate Time Series Imputation. In AAAI, 35(10):8983–8991, May 2021.

class pypots.nn.modules.usgan.BackboneUSGAN(n_steps, n_features, rnn_hidden_size, lambda_mse, hint_rate=0.7, dropout_rate=0.0)[source]¶

USGAN model

forward(inputs, training_object='generator')[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

pypots.nn.modules.vader¶

The package including the modules of VaDER.

Refer to the paper Johann de Jong, Mohammad Asif Emon, Ping Wu, Reagon Karki, Meemansa Sood, Patrice Godard, Ashar Ahmad, Henri Vrooman, Martin Hofmann-Apitius, and Holger Fröhlich. Deep learning for clustering of multivariate clinical patient trajectories with missing values. GigaScience, 8(11):giz134, November 2019.

Notes

This implementation is inspired by the official one https://github.com/johanndejong/VaDER

class pypots.nn.modules.vader.BackboneVaDER(n_steps, d_input, n_clusters, d_rnn_hidden, d_mu_stddev, eps=1e-09, alpha=1.0)[source]¶

Parameters:

n_steps (int)
d_input (int)
n_clusters (int)
d_rnn_hidden (int)
d_mu_stddev (int)
eps (float)
alpha (float) – Weight of the latent loss. The final loss = `alpha`*latent loss + reconstruction loss

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor]

Note

class pypots.nn.modules.vader.PeepholeLSTMCell(input_size, hidden_size, bias=True)[source]¶

Notes

This implementation is adapted from https://gist.github.com/Kaixhin/57901e91e5c5a8bac3eb0cbbdd3aba81

forward(X, hx=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.vader.ImplicitImputation(d_input)[source]¶

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

class pypots.nn.modules.vader.GMMLayer(d_hidden, n_clusters)[source]¶

forward()[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor]

Note