pypots.nn¶

pypots.nn.functional¶

pypots.nn.functional.gather_listed_dicts(dict_list)[source]¶

Gather batched dict output from model forward

Parameters:: dict_list (list) – A list of dict output from model forward. Each dict should have the same keys.
Returns:: A dict with the same keys as the input dict, but with values concatenated along the batch dimension.
Return type:: gathered_dict

pypots.nn.functional.nonstationary_norm(X, missing_mask=None)[source]¶

Normalization from Non-stationary Transformer. Please refer to [30] for more details.

Parameters:

X (torch.Tensor) – Input data to be normalized. Shape: (n_samples, n_steps (seq_len), n_features).
missing_mask (torch.Tensor, optional) – Missing mask has the same shape as X. 1 indicates observed and 0 indicates missing.

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

X_enc (torch.Tensor) – Normalized data. Shape: (n_samples, n_steps (seq_len), n_features).
means (torch.Tensor) – Means values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).
stdev (torch.Tensor) – Standard deviation values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).

pypots.nn.functional.nonstationary_denorm(X, means, stdev)[source]¶

De-Normalization from Non-stationary Transformer. Please refer to [30] for more details.

Parameters:

X (torch.Tensor) – Input data to be de-normalized. Shape: (n_samples, n_steps (seq_len), n_features).
means (torch.Tensor) – Means values for de-normalization . Shape: (n_samples, n_features) or (n_samples, 1, n_features).
stdev (torch.Tensor) – Standard deviation values for de-normalization. Shape: (n_samples, n_features) or (n_samples, 1, n_features).

Returns:

X_denorm – De-normalized data. Shape: (n_samples, n_steps (seq_len), n_features).

Return type:

torch.Tensor

pypots.nn.functional.calc_mae(predictions, targets, masks=None)[source]¶

Calculate the Mean Absolute Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_mae
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> mae = calc_mae(predictions, targets)

mae = 0.6 here, the error is from the 3rd and 5th elements and is $|3-1|+|5-6|=3$ , so the result is 3/5=0.6.

If we want to prevent some values from MAE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> mae = calc_mae(predictions, targets, masks)

mae = 0.5 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|=1$ , so the result is 1/2=0.5.

pypots.nn.functional.calc_mse(predictions, targets, masks=None)[source]¶

Calculate the Mean Square Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_mse
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> mse = calc_mse(predictions, targets)

mse = 1 here, the error is from the 3rd and 5th elements and is $|3-1|^2+|5-6|^2=5$ , so the result is 5/5=1.

If we want to prevent some values from MSE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> mse = calc_mse(predictions, targets, masks)

mse = 0.5 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|^2=1$ , so the result is 1/2=0.5.

pypots.nn.functional.calc_rmse(predictions, targets, masks=None)[source]¶

Calculate the Root Mean Square Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_rmse
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> rmse = calc_rmse(predictions, targets)

rmse = 1 here, the error is from the 3rd and 5th elements and is $|3-1|^2+|5-6|^2=5$ , so the result is $\sqrt{5/5}=1$ .

If we want to prevent some values from RMSE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> rmse = calc_rmse(predictions, targets, masks)

rmse = 0.707 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|^2=1$ , so the result is $\sqrt{1/2}=0.5$ .

pypots.nn.functional.calc_mre(predictions, targets, masks=None)[source]¶

Calculate the Mean Relative Error between predictions and targets. masks can be used for filtering. For values==0 in masks, values at their corresponding positions in predictions will be ignored.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor, None]) – The masks for filtering the specific values in inputs and target from evaluation. When given, only values at corresponding positions where values ==1 in masks will be used for evaluation.

Return type:

Union[float, Tensor]

Examples

>>> import numpy as np
>>> from pypots.nn.functional import calc_mre
>>> targets = np.array([1, 2, 3, 4, 5])
>>> predictions = np.array([1, 2, 1, 4, 6])
>>> mre = calc_mre(predictions, targets)

mre = 0.2 here, the error is from the 3rd and 5th elements and is $|3-1|+|5-6|=3$ , so the result is $\sqrt{3/(1+2+3+4+5)}=1$ .

If we want to prevent some values from MRE calculation, e.g. the first three elements here, we can use masks to filter out them:

>>> masks = np.array([0, 0, 0, 1, 1])
>>> mre = calc_mre(predictions, targets, masks)

mre = 0.111 here, the first three elements are ignored, the error is from the 5th element and is $|5-6|^2=1$ , so the result is $\sqrt{1/2}=0.5$ .

pypots.nn.functional.calc_quantile_crps(predictions, targets, masks, scaler_mean=0, scaler_stddev=1)[source]¶

Continuous rank probability score for distributional predictions.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor]) – The masks for filtering the specific values in inputs and target from evaluation. Only values at corresponding positions where values ==1 in masks will be used for evaluation.
scaler_mean – Mean value of the scaler used to scale the data.
scaler_stddev – Standard deviation value of the scaler used to scale the data.

Returns:

Value of continuous rank probability score.

Return type:

CRPS

pypots.nn.functional.calc_quantile_crps_sum(predictions, targets, masks, scaler_mean=0, scaler_stddev=1)[source]¶

Sum continuous rank probability score for distributional predictions.

Parameters:

predictions (Union[ndarray, Tensor]) – The prediction data to be evaluated.
targets (Union[ndarray, Tensor]) – The target data for helping evaluate the predictions.
masks (Union[ndarray, Tensor]) – The masks for filtering the specific values in inputs and target from evaluation. Only values at corresponding positions where values ==1 in masks will be used for evaluation.
scaler_mean – Mean value of the scaler used to scale the data.
scaler_stddev – Standard deviation value of the scaler used to scale the data.

Returns:

Sum value of continuous rank probability score.

Return type:

CRPS

pypots.nn.functional.calc_binary_classification_metrics(prob_predictions, targets, pos_label=1)[source]¶

Calculate the evaluation metrics for the binary classification task, including accuracy, precision, recall, f1 score, area under ROC curve, and area under Precision-Recall curve. If targets contains multiple categories, please set the positive category as pos_label.

Parameters:

prob_predictions (ndarray) – Estimated probability predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int) – The label of the positive class. Note that pos_label is also the index used to extract binary prediction probabilities from predictions.

Returns:

A dictionary contains classification metrics and useful results:

predictions: binary categories of the prediction results;

accuracy: prediction accuracy;

precision: prediction precision;

recall: prediction recall;

f1: F1-score;

precisions: precision values of Precision-Recall curve

recalls: recall values of Precision-Recall curve

pr_auc: area under Precision-Recall curve

fprs: false positive rates of ROC curve

tprs: true positive rates of ROC curve

roc_auc: area under ROC curve

Return type:

classification_metrics

pypots.nn.functional.calc_precision_recall_f1(prob_predictions, targets, pos_label=1)[source]¶

Calculate precision, recall, and F1-score of model predictions.

Parameters:

prob_predictions (ndarray) – Estimated probability predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int, default=1) – The label of the positive class.

Return type:

Tuple[float, float, float]

Returns:

precision – The precision value of model predictions.
recall – The recall value of model predictions.
f1 – The F1 score of model predictions.

pypots.nn.functional.calc_pr_auc(prob_predictions, targets, pos_label=1)[source]¶

Calculate precisions, recalls, and area under PR curve of model predictions.

Parameters:

prob_predictions (ndarray) – Estimated probability predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int, default=1) – The label of the positive class.

Return type:

Tuple[float, ndarray, ndarray, ndarray]

Returns:

pr_auc – Value of area under Precision-Recall curve.
precisions – Precision values of Precision-Recall curve.
recalls – Recall values of Precision-Recall curve.
thresholds – Increasing thresholds on the decision function used to compute precision and recall.

pypots.nn.functional.calc_roc_auc(prob_predictions, targets, pos_label=1)[source]¶

Calculate false positive rates, true positive rates, and area under AUC curve of model predictions.

Parameters:

prob_predictions (ndarray) – Estimated probabilities/predictions returned by a decision function.
targets (ndarray) – Ground truth (correct) classification results.
pos_label (int, default=1) – The label of the positive class.

Return type:

Tuple[float, ndarray, ndarray, ndarray]

Returns:

roc_auc – The area under ROC curve.
fprs – False positive rates of ROC curve.
tprs – True positive rates of ROC curve.
thresholds – Increasing thresholds on the decision function used to compute FPR and TPR.

pypots.nn.functional.calc_acc(class_predictions, targets)[source]¶

Calculate accuracy score of model predictions.

Parameters:

class_predictions (ndarray) – Estimated classification predictions returned by a classifier.
targets (ndarray) – Ground truth (correct) classification results.

Returns:

The accuracy of model predictions.

Return type:

acc_score

pypots.nn.functional.calc_rand_index(class_predictions, targets)[source]¶

Calculate Rand Index, a measure of the similarity between two data clusterings. Refer to [54].

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

Rand index.

Return type:

References

pypots.nn.functional.calc_adjusted_rand_index(class_predictions, targets)[source]¶

Calculate adjusted Rand Index.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

Adjusted Rand index.

Return type:

aRI

References

pypots.nn.functional.calc_cluster_purity(class_predictions, targets)[source]¶

Calculate cluster purity.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

cluster purity.

Return type:

cluster_purity

Notes

This function is from the answer https://stackoverflow.com/a/51672699 on StackOverflow.

pypots.nn.functional.calc_nmi(class_predictions, targets)[source]¶

Calculate Normalized Mutual Information between two clusterings.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

NMI – Normalized Mutual Information

Return type:

float,

pypots.nn.functional.calc_chs(X, predicted_labels)[source]¶

Compute the Calinski and Harabasz score (also known as the Variance Ratio Criterion).

Xarray-like of shape (n_samples_a, n_features): A feature array, or learned latent representation, that can be used for clustering.
predicted_labelsarray-like of shape (n_samples): Predicted labels for each sample.

Returns:: calinski_harabasz_score – The resulting Calinski-Harabasz score. In short, the higher, the better.
Return type:: float

References

pypots.nn.functional.calc_dbs(X, predicted_labels)[source]¶

Compute the Davies-Bouldin score.

Parameters:

X (array-like of shape (n_samples_a, n_features)) – A feature array, or learned latent representation, that can be used for clustering.
predicted_labels (array-like of shape (n_samples)) – Predicted labels for each sample.

Returns:

davies_bouldin_score – The resulting Davies-Bouldin score. In short, the lower, the better.

Return type:

float

References

pypots.nn.functional.calc_silhouette(X, predicted_labels)[source]¶

Compute the mean Silhouette Coefficient of all samples.

Parameters:

X (array-like of shape (n_samples_a, n_features)) – A feature array, or learned latent representation, that can be used for clustering.
predicted_labels (array-like of shape (n_samples)) – Predicted labels for each sample.

Returns:

silhouette_score – Mean Silhouette Coefficient for all samples. In short, the higher, the better.

Return type:

float

References

pypots.nn.functional.calc_internal_cluster_validation_metrics(X, predicted_labels)[source]¶

Computer all internal cluster validation metrics available in PyPOTS and return as a dictionary.

Parameters:

X (array-like of shape (n_samples_a, n_features)) – A feature array, or learned latent representation, that can be used for clustering.
predicted_labels (array-like of shape (n_samples)) – Predicted labels for each sample.

Returns:

internal_cluster_validation_metrics – A dictionary contains all internal cluster validation metrics available in PyPOTS.

Return type:

dict

pypots.nn.functional.calc_external_cluster_validation_metrics(class_predictions, targets)[source]¶

Computer all external cluster validation metrics available in PyPOTS and return as a dictionary.

Parameters:

class_predictions (ndarray) – Clustering results returned by a clusterer.
targets (ndarray) – Ground truth (correct) clustering results.

Returns:

external_cluster_validation_metrics – A dictionary contains all external cluster validation metrics available in PyPOTS.

Return type:

dict

pypots.nn.modules.base_model_core¶

class pypots.nn.modules.base_model_core.ModelCore(*args, **kwargs)[source]¶

Base class for all NN models’ core. For most of the NN models (inheriting BaseNNModel) in PyPOTS, their model core inherits this class and implement the forward and calc_criterion methods, and they exempt from overwriting _train_model() which carries concrete model training details, e.g. SAITS, Transformer, and this is task agnostic. But some models’ processing procedures are too complex to be abstracted into this class, e.g. GAN models (like US-GAN and CRLI) which have more than one optimizer need to update or VaDER which needs pretraining and has complicated manipulations for its Gaussian mixture model, their model core don’t have to inherit this class, just be a subclass of torch.nn.Module and obey basic rules (include the loss/metric as keys in the return dict) to be compatible with PyPOTS. Additionally, in this case, they need to overwrite _train_model() to implement their own training procedures.

abstract forward(inputs, calc_criterion=False)[source]¶

Forward pass of the model. This method should be implemented to include all operations in the NN model’s forward propagation.

Parameters:

inputs (dict) – The dictionary including all necessary input data for the model’s forward pass.
calc_criterion (bool) – Whether to calculate training loss (when self.training==True) and validation metric (when self.training!=True), and pass the criterion out together with the model forward results.

Returns:

The model’s output results of its forward propagation.

Return type:

dict

pypots.nn.modules.loss¶

class pypots.nn.modules.loss.Criterion(lower_better=True)[source]¶

forward(logits, targets)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.loss.MSE[source]¶

forward(logits, targets, masks=None)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.loss.MAE[source]¶

forward(logits, targets, masks=None)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.loss.RMSE[source]¶

forward(logits, targets, masks=None)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.loss.MRE[source]¶

forward(logits, targets, masks=None)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.loss.CrossEntropy[source]¶

forward(logits, targets)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.loss.NLL[source]¶

forward(log_probs, targets)[source]¶

The criterion calculation process.

Parameters:

logits – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

pypots.nn.modules.metric¶

class pypots.nn.modules.metric.PR_AUC(pos_label=1)[source]¶

forward(logits, targets)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.metric.ROC_AUC(pos_label=1)[source]¶

forward(logits, targets)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

class pypots.nn.modules.metric.Accuracy[source]¶

forward(logits, targets)[source]¶

The criterion calculation process.

Parameters:

logits (Tensor) – The model outputs, predicted unnormalized logits.
targets (Tensor) – The ground truth values.

Return type:

Tensor

pypots.nn.modules.revin¶

The package including the modules of RevIN.

Refer to the paper Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. “Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift”. In International Conference on Learning Representations, 2022.

Notes

This implementation is inspired by the official one https://github.com/ts-kim/RevIN

class pypots.nn.modules.revin.RevIN(n_features, eps=1e-09, affine=True)[source]¶

RevIN: Reversible Inference Network.

Parameters:

n_features (int) – the number of features or channels
eps (float) – a value added for numerical stability
affine (bool) – if True, RevIN has learnable affine parameters

forward(x, missing_mask=None, mode='norm')[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pypots.nn.modules.tcn¶

The package including the modules of TCN (Temporal Convolutional Network).

Refer to the paper Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. “An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling”. arXiv preprint arXiv:1803.01271.

Notes

This implementation is inspired by the official one https://github.com/locuslab/TCN

class pypots.nn.modules.tcn.BackboneTCN(num_inputs, num_channels, kernel_size=2, dropout=0.2)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.pyraformer¶

The package including the modules of Pyraformer.

Refer to the paper Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. “Pyraformer: Low-Complexity Pyramidal Attention for Long-Range Time Series Modeling and Forecasting”. International Conference on Learning Representations. 2022.

Notes

This implementation is inspired by the official one https://github.com/ant-research/Pyraformer

class pypots.nn.modules.pyraformer.PyraformerEncoder(n_steps, n_layers, d_model, n_heads, d_ffn, dropout, attn_dropout, window_size, inner_size)[source]¶

forward(x, src_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Union[Tensor, Tuple[Tensor, list]]

Note

pypots.nn.modules.film¶

The package including the modules of FiLM.

Refer to the paper Zhou, Tian, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, and Rong Jin. “Film: Frequency improved legendre memory model for long-term time series forecasting.” In Advances in Neural Information Processing Systems 35 (2022): 12677-12690.

Notes

This implementation is inspired by the official one https://github.com/tianzhou2011/FiLM

class pypots.nn.modules.film.HiPPO_LegT(N, dt=1.0, discretization='bilinear')[source]¶

forward(inputs)[source]¶: inputs : (length, …) output : (length, …, N) where N is the order of the HiPPO projection

class pypots.nn.modules.film.SpectralConv1d(in_channels, out_channels, seq_len, modes1, ratio=0.5, mode_type=0)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.film.BackboneFiLM(n_steps, in_channels, n_pred_steps, window_size, multiscale, modes1, ratio, mode_type)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

pypots.nn.modules.raindrop¶

The package including the modules of Raindrop.

Refer to the paper Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In ICLR, 2022.

Notes

This implementation is inspired by the official one the official implementation https://github.com/mims-harvard/Raindrop

class pypots.nn.modules.raindrop.BackboneRaindrop(n_features, n_layers, d_model, n_heads, d_ffn, n_classes, dropout=0.3, max_len=215, d_static=9, d_pe=16, aggregation='mean', sensor_wise_mask=False, static=False)[source]¶

forward(X, timestamps, lengths)[source]¶

Forward processing of Raindrop.

Parameters:

X (Tensor) – The input tensor of shape (batch_size, n_features, max_len).
timestamps (Tensor) – The timestamps tensor of shape (batch_size, max_len).
lengths (Tensor) – The lengths tensor of shape (batch_size, 1).

Returns:

prediction

Return type:

torch.Tensor

pypots.nn.modules.frets¶

The package including the modules of FiLM.

Refer to the paper Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. “Frequency-domain MLPs are More Effective Learners in Time Series Forecasting.” Advances in Neural Information Processing Systems 36 (2024).

Notes

This implementation is inspired by the official one https://github.com/aikunyi/FreTS

class pypots.nn.modules.frets.BackboneFreTS(n_steps, n_features, embed_size, n_pred_steps, hidden_size, channel_independence=False)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.timesnet¶

The package including the modules of TimesNet.

Refer to the paper Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/thuml/Time-Series-Library

class pypots.nn.modules.timesnet.BackboneTimesNet(n_layers, n_steps, n_pred_steps, top_k, d_model, d_ffn, n_kernels)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

class pypots.nn.modules.timesnet.TimesBlock(seq_len, pred_len, top_k, d_model, d_ffn, num_kernels)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.moment¶

The package including the modules of MOMENT.

Refer to the paper Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. “MOMENT: A Family of Open Time-series Foundation Models”. In ICML, 2024.

Notes

This implementation is inspired by the official one https://github.com/moment-timeseries-foundation-model/moment-research

class pypots.nn.modules.moment.BackboneMOMENT(configs, **kwargs)[source]¶

embed(x_enc, input_mask=None, reduction='mean', **kwargs)[source]¶

x_enc : [batch_size x n_channels x seq_len] input_mask : [batch_size x 1 x seq_len]

Return type:: TimeseriesOutputs

pretraining(x_enc, input_mask=None, mask=None, **kwargs)[source]¶

x_enc[batch_size x n_channels x seq_len]: Time-series data
mask[batch_size x seq_len]: Data that is masked but still attended to via mask-tokens
input_mask[batch_size x seq_len]: Input mask for the time-series data that is unobserved. This is typically padded data, that is not attended to.

reconstruct(x_enc, input_mask=None, mask=None, **kwargs)[source]¶

x_enc[batch_size x n_channels x seq_len]: Time-series data
mask[batch_size x seq_len]: Data that is masked but still attended to via mask-tokens
input_mask[batch_size x seq_len]: Input mask for the time-series data that is unobserved. This is typically padded data, that is not attended to.

detect_anomalies(x_enc, input_mask=None, anomaly_criterion='mse', **kwargs)[source]¶: x_enc : [batch_size x n_channels x seq_len] input_mask : [batch_size x seq_len] anomaly_criterion : str

long_forecast(x_enc, input_mask=None, **kwargs)[source]¶: x_enc : [batch_size x n_channels x seq_len] input_mask : [batch_size x seq_len]

short_forecast(x_enc, input_mask=None, forecast_horizon=1, **kwargs)[source]¶: x_enc : [batch_size x n_channels x seq_len] input_mask : [batch_size x seq_len] forecast_horizon : int

forward(x_enc, mask=None, input_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.inception¶

The package including the modules of Inception model.

Refer to the paper Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich. Going deeper with convolutions. CVPR 2015.

class pypots.nn.modules.inception.InceptionBlockV1(in_channels, out_channels, num_kernels=6, stride=1, init_weight=True)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.inception.InceptionTransBlockV1(in_channels, out_channels, stride=1, num_kernels=6, init_weight=True)[source]¶

forward(x, output_size)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.segrnn¶

The package including the modules of SegRNN.

Refer to the paper Lin, Shengsheng and Lin, Weiwei and Wu, Wentai and Zhao, Feiyu and Mo, Ruichao and Zhang, Haotong. Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308.11200.

Notes

This implementation is inspired by the official one https://github.com/lss-1138/SegRNN

class pypots.nn.modules.segrnn.BackboneSegRNN(n_steps, n_features, n_pred_steps, seg_len=24, d_model=512, dropout=0.5)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.stemgnn¶

The package including the modules of StemGNN.

Refer to the paper Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, Qi Zhang. “Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting”. In Advances in Neural Information Processing Systems, 2020.

Notes

This implementation is inspired by the official one https://github.com/microsoft/StemGNN

class pypots.nn.modules.stemgnn.BackboneStemGNN(units, stack_cnt, time_step, multi_layer, horizon=1, dropout_rate=0.5, leaky_rate=0.2)[source]¶

static get_laplacian(graph, normalize)[source]¶: return the laplacian of the graph. :type graph: :param graph: the graph structure without self loop, [N, N]. :type normalize: :param normalize: whether to used the normalized laplacian. :return: graph laplacian.

static cheb_polynomial(laplacian)[source]¶: Compute the Chebyshev Polynomial, according to the graph laplacian. :type laplacian: :param laplacian: the graph laplacian, [N, N]. :return: the multi order Chebyshev laplacian, [K, N, N].

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.timellm¶

The package including the modules of Time-LLM.

Refer to the paper Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. In the 12th International Conference on Learning Representations, 2024.

Notes

This implementation is inspired by the official one https://github.com/KimMeen/Time-LLM

class pypots.nn.modules.timellm.BackboneTimeLLM(n_steps, n_features, n_pred_steps, n_layers, patch_size, patch_stride, d_model, d_ffn, d_llm, n_heads, llm_model_type, dropout, domain_prompt_content, task_name)[source]¶

forward(x_enc, missing_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.imputeformer¶

The package including the modules of ImputeFormer.

Refer to the paper Tong Nie, Guoyang Qin, Wei Ma, Yuewen Mei, Jian Sun. ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation. KDD, 2024.

Notes

This implementation is inspired by the official one https://github.com/tongnie/ImputeFormer

class pypots.nn.modules.imputeformer.EmbeddedAttentionLayer(model_dim, node_embedding_dim, feed_forward_dim=2048, dropout=0)[source]¶

forward(x, emb, dim=-2)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.imputeformer.ProjectedAttentionLayer(seq_len, dim_proj, d_model, n_heads, d_ff=None, dropout=0.1)[source]¶

Temporal projected attention layer. A low-rank factorization is achieved in the temporal attention matrix.

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.imputeformer.MLP(input_size, hidden_size, output_size=None, n_layers=1, dropout=0.0)[source]¶: Simple Multi-layer Perceptron encoder with optional linear readout.

pypots.nn.modules.csai¶

The package including the modules of CSAI.

Refer to the paper Linglong Qian, Zina Ibrahim, Hugh Logan Ellis, Ao Zhang, Yuezhou Zhang, Tao Wang, Richard Dobson. Knowledge Enhanced Conditional Imputation for Healthcare Time-series. In Arxiv, 2024.

Notes

This implementation is inspired by the official one the official implementation https://github.com/LinglongQian/CSAI.

class pypots.nn.modules.csai.BackboneCSAI(n_steps, n_features, rnn_hidden_size, step_channels, training_loss=MAE())[source]¶

Attributes:

n_steps – sequence length (number of time steps)
n_features – number of features (input dimensions)
rnn_hidden_size – the hidden size of the GRU cell
step_channels – number of channels for each step in the sequence
medians_tensor – tensor of median values for features, used to adjust decayed observations
temp_decay_h – the temporal decay module to decay the hidden state of the GRU
temp_decay_x – the temporal decay module to decay data in the raw feature space
hist – the temporal-regression module that projects the GRU hidden state into the raw feature space
feat_reg_v – the feature-regression module used for feature-based estimation
weight_combine – the module that generates the weight to combine history regression and feature regression
weighted_obs – the decay module that computes weighted decay based on observed data and deltas
gru – the GRU cell that models temporal data for imputation
pos_encoder – the positional encoding module that adds temporal information to the sequence data
input_projection – the convolutional module used to project input features into a higher-dimensional space
output_projection1 – the convolutional module used to project the output from the Transformer layer
output_projection2 – the final convolutional module used to generate the hidden state from the time-layer’s output
time_layer – the Transformer encoder layer used to model complex temporal dependencies within the sequence
device – the device (CPU/GPU) used for model computations

Parameters:

n_steps – sequence length (number of time steps)
n_features – number of features (input dimensions)
rnn_hidden_size – the hidden size of the GRU cell
step_channels – number of channels for each step in the sequence

forward(x, mask, deltas, last_obs, intervals, h=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csai.BackboneBCSAI(n_steps, n_features, rnn_hidden_size, step_channels, training_loss=MAE())[source]¶

forward(xdata)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csai.FeatureRegression(input_size)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csai.Decay(input_size, output_size, diag=False)[source]¶

forward(d)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csai.Decay_obs(input_size, output_size)[source]¶

forward(delta_diff)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csai.PositionalEncoding(d_model, dropout=0.1, max_len=5000)[source]¶

forward(x)[source]¶

Parameters:: x – Tensor, shape [seq_len, batch_size, embedding_dim]

class pypots.nn.modules.csai.Conv1dWithInit(in_channels, out_channels, kernel_size)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csai.TorchTransformerEncoder(heads=8, layers=1, channels=64)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.transformer¶

The package including the modules of Transformer.

Refer to the papers Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. and Wenjie Du, David Cote, and Yan Liu. SAITS: Self-Attention-based Imputation for Time Series. Expert Systems with Applications, 219:119619, 2023.

Notes

This implementation is inspired by https://github.com/WenjieDu/SAITS

class pypots.nn.modules.transformer.ScaledDotProductAttention(temperature, attn_dropout=0.1)[source]¶

Scaled dot-product attention.

Parameters:

temperature (float) – The temperature for scaling.
attn_dropout (float) – The dropout rate for the attention map.

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Forward processing of the scaled dot-product attention.

Parameters:

q (Tensor) – Query tensor.
k (Tensor) – Key tensor.
v (Tensor) – Value tensor.
attn_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps]. 0 in attn_mask means values at the according position in the attention map will be masked out.

Return type:

Tuple[Tensor, Tensor]

Returns:

output – The result of Value multiplied with the scaled dot-product attention map.
attn – The scaled dot-product attention map.

class pypots.nn.modules.transformer.MultiHeadAttention(attn_opt, d_model, n_heads, d_k, d_v)[source]¶

Transformer multi-head attention module.

Parameters:

attn_opt (AttentionOperator) – The attention operator, e.g. the self-attention proposed in Transformer.
d_model (int) – The dimension of the input tensor.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.

forward(q, k, v, attn_mask, **kwargs)[source]¶

Forward processing of the multi-head attention module.

Parameters:

q (Tensor) – Query tensor.
k (Tensor) – Key tensor.
v (Tensor) – Value tensor.
attn_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps]. 0 in attn_mask means values at the according position in the attention map will be masked out.

Return type:

Tuple[Tensor, Tensor]

Returns:

v – The output of the multi-head attention layer.
attn_weights – The attention map.

class pypots.nn.modules.transformer.PositionalEncoding(d_hid, n_positions=1000)[source]¶

The original positional-encoding module for Transformer.

Parameters:

d_hid (int) – The dimension of the hidden layer.
n_positions (int) – The max number of positions.

forward(x, dim=1, return_only_pos=False)[source]¶

Forward processing of the positional encoding module.

Parameters:

x (Tensor) – Input tensor.
dim (int) – The dimension to add the positional encoding.
return_only_pos (bool) – Whether to return only the positional encoding.

Return type:

Tensor

Returns:

If return_only_pos is True –

pos_enc:
The positional encoding.
else –

x_with_pos:
Output tensor, the input tensor with the positional encoding added.

class pypots.nn.modules.transformer.TransformerEncoderLayer(attn_opt, d_model, n_heads, d_k, d_v, d_ffn, dropout=0.1)[source]¶

Transformer encoder layer.

Parameters:

attn_opt (AttentionOperator) – The attention operator for the multi-head attention module in the encoder layer.
d_model (int) – The dimension of the input tensor.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer.
dropout (float) – The dropout rate.

forward(enc_input, src_mask=None, **kwargs)[source]¶

Forward processing of the encoder layer.

Parameters:

enc_input (Tensor) – Input tensor.
src_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Tuple[Tensor, Tensor]

Returns:

enc_output – Output tensor.
attn_weights – The attention map.

class pypots.nn.modules.transformer.TransformerDecoderLayer(slf_attn_opt, enc_attn_opt, d_model, n_heads, d_k, d_v, d_ffn, dropout=0.1)[source]¶

Transformer decoder layer.

Parameters:

slf_attn_opt (AttentionOperator) – The attention operator for the multi-head attention module in the decoder layer.
enc_attn_opt (AttentionOperator) – The attention operator for the encoding multi-head attention module in the decoder layer.
d_model (int) – The dimension of the input tensor.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer.
dropout (float) – The dropout rate.

forward(dec_input, enc_output, slf_attn_mask=None, dec_enc_attn_mask=None, **kwargs)[source]¶

Forward processing of the decoder layer.

Parameters:

dec_input (Tensor) – Input tensor.
enc_output (Tensor) – Output tensor from the encoder.
slf_attn_mask (Optional[Tensor]) – Masking tensor for the self-attention module. The shape should be [batch_size, n_heads, n_steps, n_steps].
dec_enc_attn_mask (Optional[Tensor]) – Masking tensor for the encoding attention module. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

dec_output – Output tensor.
dec_slf_attn – The self-attention map.
dec_enc_attn – The encoding attention map.

class pypots.nn.modules.transformer.PositionWiseFeedForward(d_in, d_hid, dropout=0.1)[source]¶

Position-wise feed forward network (FFN) in Transformer.

Parameters:

d_in (int) – The dimension of the input tensor.
d_hid (int) – The dimension of the hidden layer.
dropout (float) – The dropout rate.

forward(x)[source]¶

Forward processing of the position-wise feed forward network.

Parameters:: x (Tensor) – Input tensor.
Returns:: Output tensor.
Return type:: x

class pypots.nn.modules.transformer.TransformerEncoder(n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

Transformer encoder.

Parameters:

n_layers (int) – The number of layers in the encoder.
d_model (int) – The dimension of the module manipulation space. The input tensor will be projected to a space with d_model dimensions.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer in the feed-forward network.
dropout (float) – The dropout rate.
attn_dropout (float) – The dropout rate for the attention map.

forward(x, src_mask=None)[source]¶

Forward processing of the encoder.

Parameters:

x (Tensor) – Input tensor.
src_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Union[Tensor, Tuple[Tensor, list]]

Returns:

enc_output – Output tensor.
attn_weights_collector – A list containing the attention map from each encoder layer.

class pypots.nn.modules.transformer.TransformerDecoder(n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

Transformer decoder.

Parameters:

n_layers (int) – The number of layers in the decoder.
d_model (int) – The dimension of the module manipulation space. The input tensor will be projected to a space with d_model dimensions.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer in the feed-forward network.
dropout (float) – The dropout rate.
attn_dropout (float) – The dropout rate for the attention map.

forward(trg_seq, enc_output, trg_mask=None, src_mask=None)[source]¶

Forward processing of the decoder.

Parameters:

trg_seq (Tensor) – Input tensor.
enc_output (Tensor) – Output tensor from the encoder.
trg_mask (Optional[Tensor]) – Masking tensor for the self-attention module.
src_mask (Optional[Tensor]) – Masking tensor for the encoding attention module.

Return type:

Union[Tensor, Tuple[Tensor, list, list]]

Returns:

dec_output – Output tensor.
dec_slf_attn_collector – A list containing the self-attention map from each decoder layer.
dec_enc_attn_collector – A list containing the encoding attention map from each decoder layer.

pypots.nn.modules.mrnn¶

The package including the modules of M-RNN.

Refer to the paper Jinsung Yoon, William R. Zame, and Mihaela van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 66(5):14771490, 2019.

Notes

This implementation is inspired by the official one https://github.com/jsyoon0823/MRNN and https://github.com/WenjieDu/SAITS

class pypots.nn.modules.mrnn.BackboneMRNN(n_steps, n_features, rnn_hidden_size)[source]¶

forward(inputs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor]

Note

class pypots.nn.modules.mrnn.MrnnFcnRegression(feature_num)[source]¶

M-RNN fully connection regression Layer

forward(x, missing_mask, target)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.reformer¶

The package including the modules of Reformer.

Refer to the paper Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. “Reformer: The Efficient Transformer”. In International Conference on Learning Representations, 2020.

Notes

This implementation is inspired by the official one https://github.com/google/trax/tree/master/trax/models/reformer and https://github.com/lucidrains/reformer-pytorch

class pypots.nn.modules.reformer.ReformerEncoder(n_steps, n_layers, d_model, n_heads, bucket_size, n_hashes, causal, d_ffn, dropout)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.micn¶

The package including the modules of MICN.

Refer to the paper Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao “MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting”. In the Eleventh International Conference on Learning Representations, 2023.

Notes

This implementation is inspired by the official one https://github.com/wanghq21/MICN

class pypots.nn.modules.micn.BackboneMICN(n_steps, n_features, n_pred_steps, n_pred_features, n_layers, d_model, decomp_kernel, isometric_kernel, conv_kernel)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.trmf¶

class pypots.nn.modules.trmf.BackboneTRMF(lags, K, lambda_f, lambda_x, lambda_w, alpha, eta, max_iter, F_step=0.0001, X_step=0.0001, W_step=0.0001)[source]¶

The backbone of Temporal Regularized Matrix Factorization (TRMF).

Parameters:

lags (array-like, shape (n_lags,)) – Set of lag indices to use in model.
K (int) – Length of latent embedding dimension
lambda_f (float) – Regularization parameter used for matrix F.
lambda_x (float) – Regularization parameter used for matrix X.
lambda_w (float) – Regularization parameter used for matrix W.
alpha (float) – Regularization parameter used for make the sum of lag coefficient close to 1. That helps to avoid big deviations when forecasting.
eta (float) – Regularization parameter used for X when undercovering autoregressive dependencies.
max_iter (int) – Number of iterations of updating matrices F, X and W.
F_step (float) – Step of gradient descent when updating matrix F.
X_step (float) – Step of gradient descent when updating matrix X.
W_step (float) – Step of gradient descent when updating matrix W.

Attributes:

F (ndarray, shape (n_timeseries, K)) – Latent embedding of timeseries.
X (ndarray, shape (K, n_timepoints)) – Latent embedding of timepoints.
W (ndarray, shape (K, n_lags)) – Matrix of autoregressive coefficients.

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

impute_missingness()[source]¶

Impute each missing element in timeseries.

Model uses matrix X and F to get all missing elements.

Returns:: data – Imputed data.
Return type:: ndarray, shape (n_timeseries, T)

pypots.nn.modules.tefn¶

The package of the forecasting model TEFN.

Refer to the paper Tianxiang Zhan, Yuanpeng He, Yong Deng, and Zhen Li. Time Evidence Fusion Network: Multi-source View in Long-Term Time Series Forecasting. In Arxiv, 2024.

Notes

This implementation is transfered from the official one https://github.com/ztxtech/Time-Evidence-Fusion-Network

class pypots.nn.modules.tefn.BackboneTEFN(n_steps, n_features, n_pred_steps, n_fod)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

pypots.nn.modules.informer¶

The package including the modules of Informer.

Refer to the paper Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115, 2021.

Notes

This implementation is inspired by the official one https://github.com/zhouhaoyi/Informer2020

class pypots.nn.modules.informer.ProbAttention(mask_flag=True, factor=5, attention_dropout=0.1, scale=None)[source]¶

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.ConvLayer(c_in)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerEncoderLayer(attention, d_model, d_ff=None, dropout=0.1, activation='relu')[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerDecoderLayer(self_attention, cross_attention, d_model, d_ff=None, dropout=0.1, activation='relu')[source]¶

forward(x, cross, x_mask=None, cross_mask=None, tau=None, delta=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerEncoder(attn_layers, conv_layers=None, norm_layer=None)[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.informer.InformerDecoder(layers, norm_layer=None, projection=None)[source]¶

forward(x, cross, x_mask=None, cross_mask=None, trend=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.dlinear¶

The package including the modules of DLinear.

Refer to the paper Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, volume 37, pages 11121–11128, Jun. 2023.

Notes

This implementation is inspired by the official one https://github.com/cure-lab/LTSF-Linear

class pypots.nn.modules.dlinear.BackboneDLinear(n_steps, n_features, individual=False, d_model=None)[source]¶

forward(seasonal_init, trend_init)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.etsformer¶

The package including the modules of ETSformer.

Refer to the paper Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. ETSformer: Exponential smoothing transformers for time-series forecasting. In ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/salesforce/ETSformer

class pypots.nn.modules.etsformer.ETSformerEncoder(layers)[source]¶

forward(res, level, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.etsformer.ETSformerEncoderLayer(d_model, n_heads, d_out, seq_len, pred_len, k, d_ffn=None, dropout=0.1, activation='sigmoid', layer_norm_eps=1e-05)[source]¶

forward(res, level, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.etsformer.ETSformerDecoder(layers)[source]¶

forward(growths, seasons)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.etsformer.ETSformerDecoderLayer(d_model, n_heads, d_out, pred_len, dropout=0.1)[source]¶

forward(growth, season)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.fits¶

The package including the modules of FITS.

Refer to the paper Zhijian Xu, Ailing Zeng, and Qiang Xu. FITS: Modeling Time Series with 10k parameters. In The Twelfth International Conference on Learning Representations, 2024.

Notes

This implementation is inspired by the official one https://github.com/VEWOXIC/FITS

class pypots.nn.modules.fits.BackboneFITS(n_steps, n_features, n_pred_steps, cut_freq, individual)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.autoformer¶

The package including the modules of Autoformer.

Refer to the paper Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting. In Advances in Neural Information Processing Systems, volume 34, pages 22419–22430. Curran Associates, Inc., 2021.

Notes

This implementation is inspired by the official one https://github.com/thuml/Autoformer

class pypots.nn.modules.autoformer.AutoCorrelation(factor=1, attention_dropout=0.1)[source]¶

AutoCorrelation Mechanism with the following two phases:

period-based dependencies discovery
time delay aggregation

This block can replace the self-attention family mechanism seamlessly.

time_delay_agg_training(values, corr)[source]¶: SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the training phase.

time_delay_agg_inference(values, corr)[source]¶: SpeedUp version of Autocorrelation (a batch-normalization style design) This is for the inference phase.

time_delay_agg_full(values, corr)[source]¶: Standard version of Autocorrelation

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.autoformer.SeasonalLayerNorm(n_channels)[source]¶

A special designed layer normalization for the seasonal part.

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.MovingAvgBlock(kernel_size, stride)[source]¶

The moving average block to highlight the trend of time series.

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.SeriesDecompositionBlock(kernel_size)[source]¶

Series decomposition block

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.AutoformerEncoderLayer(attn_opt, d_model, n_heads, d_ffn, moving_avg=25, dropout=0.1, activation='relu')[source]¶

Autoformer encoder layer with the progressive decomposition architecture.

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.AutoformerDecoderLayer(self_attn_opt, cross_attn_opt, d_model, n_heads, d_out, d_ff=None, moving_avg=25, dropout=0.1, activation='relu')[source]¶

Autoformer decoder layer with the progressive decomposition architecture

forward(x, cross, x_mask=None, cross_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.autoformer.AutoformerEncoder(n_layers, d_model, n_heads, d_ffn, factor, moving_avg_window_size, dropout, activation='relu')[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.scinet¶

The package including the modules of SCINet.

Refer to the paper Minhao LIU, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia LAI, Lingna Ma, and Qiang Xu. “SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction”. In Advances in Neural Information Processing Systems, 2022.

Notes

This implementation is inspired by the official one https://github.com/cure-lab/SCINet

class pypots.nn.modules.scinet.BackboneSCINet(n_out_steps, n_in_steps, n_in_features, d_hidden, n_stacks, n_levels, n_decoder_layers, n_groups, kernel_size=5, dropout=0.5, concat_len=0, pos_enc=False, modified=True, single_step_output_One=False)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.brits¶

The package including the modules of BRITS.

Refer to the paper Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. BRITS: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

Notes

This implementation is inspired by the official one https://github.com/caow13/BRITS The bugs in the original implementation are fixed here.

class pypots.nn.modules.brits.BackboneRITS(n_steps, n_features, rnn_hidden_size, training_loss=MAE())[source]¶

model RITS: Recurrent Imputation for Time Series

Attributes:

n_steps – sequence length (number of time steps)
n_features – number of features (input dimensions)
rnn_hidden_size – the hidden size of the RNN cell
rnn_cell – the LSTM cell to model temporal data
temp_decay_h – the temporal decay module to decay RNN hidden state
temp_decay_x – the temporal decay module to decay data in the raw feature space
hist_reg – the temporal-regression module to project RNN hidden state into the raw feature space
feat_reg – the feature-regression module
combining_weight – the module used to generate the weight to combine history regression and feature regression

Parameters:

n_steps (int) – sequence length (number of time steps)
n_features (int) – number of features (input dimensions)
rnn_hidden_size (int) – the hidden size of the RNN cell

forward(inputs, direction)[source]¶

Parameters:

inputs (dict) – Input data, a dictionary includes feature values, missing masks, and time-gap values.
direction (str) – A keyword to extract data from inputs.

Return type:

Tuple[Tensor, Tensor, Tensor, Tensor]

Returns:

imputed_data – Input data with missing parts imputed. Shape of [batch size, sequence length, feature number].
estimations – Reconstructed data. Shape of [batch size, sequence length, feature number].
hidden_states (tensor,) – [batch size, RNN hidden size]
reconstruction_loss – reconstruction loss

class pypots.nn.modules.brits.BackboneBRITS(n_steps, n_features, rnn_hidden_size, training_loss=MAE())[source]¶

model BRITS: Bidirectional RITS BRITS consists of two RITS, which take time-series data from two directions (forward/backward) respectively.

Attributes:

n_steps – sequence length (number of time steps)
n_features – number of features (input dimensions)
rnn_hidden_size – the hidden size of the RNN cell
rits_f (RITS object) – the forward RITS model
rits_b (RITS object) – the backward RITS model

forward(inputs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

class pypots.nn.modules.brits.FeatureRegression(input_size)[source]¶

The module used to capture the correlation between features for imputation in BRITS.

Attributes:

W (tensor) – The weights (parameters) of the module.
b (tensor) – The bias of the module.
m (buffer) (tensor) – The mask matrix, a squire matrix with diagonal entries all zeroes while left parts all ones. It is applied to the weight matrix to mask out the estimation contributions from features themselves. It is used to help enhance the imputation performance of the network.

Parameters:

input_size (the feature dimension of the input)

forward(x)[source]¶

Forward processing of the NN module.

Parameters:: x (tensor,) – the input for processing
Returns:: output – the processed result containing imputation from feature regression
Return type:: tensor,

pypots.nn.modules.koopa¶

The package including the modules of Koopa.

Refer to the paper Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. “Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors”. Advances in Neural Information Processing Systems 36 (2023).

Notes

This implementation is inspired by the official one https://github.com/thuml/Koopa

class pypots.nn.modules.koopa.BackboneKoopa(n_steps, n_features, n_pred_steps, n_seg_steps, d_dynamic, d_hidden, n_hidden_layers, n_blocks, multistep, alpha=0.2)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.koopa.MLP(d_in, d_out, d_hidden=128, n_hidden_layers=2, dropout=0.05, activation='tanh')[source]¶

Multilayer perceptron to encode/decode high dimension representation of sequential data

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.koopa.FourierFilter(mask_spectrum)[source]¶

Fourier Filter: to time-variant and time-invariant term

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.koopa.TimeVarKP(enc_in=8, input_len=96, pred_len=96, seg_len=24, dynamic_dim=128, encoder=None, decoder=None, multistep=False)[source]¶

Koopman Predictor with DMD (analysitical solution of Koopman operator) Utilize local variations within individual sliding window to predict the future of time-variant term

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.koopa.TimeInvKP(input_len=96, pred_len=96, dynamic_dim=128, encoder=None, decoder=None)[source]¶

Koopman Predictor with learnable Koopman operator Utilize lookback and forecast window snapshots to predict the future of time-invariant term

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.tide¶

The package including the modules of TiDE.

Refer to the paper Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. “Long-term Forecasting with TiDE: Time-series Dense Encoder”. In Transactions on Machine Learning Research, 2023.

Notes

This implementation is inspired by the official one https://github.com/google-research/google-research/blob/master/tide and https://github.com/lich99/TiDE

class pypots.nn.modules.tide.TideEncoder(n_steps, n_features, n_layers, d_flatten, d_hidden, dropout)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.tide.TideDecoder(n_steps, n_pred_steps, n_pred_features, n_layers, d_hidden, d_feature_encode, dropout)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.tide.ResBlock(input_dim, hidden_dim, output_dim, dropout=0.1, bias=True)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.vader¶

The package including the modules of VaDER.

Refer to the paper Johann de Jong, Mohammad Asif Emon, Ping Wu, Reagon Karki, Meemansa Sood, Patrice Godard, Ashar Ahmad, Henri Vrooman, Martin Hofmann-Apitius, and Holger Fröhlich. Deep learning for clustering of multivariate clinical patient trajectories with missing values. GigaScience, 8(11):giz134, November 2019.

Notes

This implementation is inspired by the official one https://github.com/johanndejong/VaDER

class pypots.nn.modules.vader.BackboneVaDER(n_steps, d_input, n_clusters, d_rnn_hidden, d_mu_stddev, eps=1e-09, alpha=1.0)[source]¶

Parameters:

n_steps (int)
d_input (int)
n_clusters (int)
d_rnn_hidden (int)
d_mu_stddev (int)
eps (float)
alpha (float) – Weight of the latent loss. The final loss = `alpha`*latent loss + reconstruction loss

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor]

Note

class pypots.nn.modules.vader.PeepholeLSTMCell(input_size, hidden_size, bias=True)[source]¶

Notes

This implementation is adapted from https://gist.github.com/Kaixhin/57901e91e5c5a8bac3eb0cbbdd3aba81

forward(X, hx=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.vader.ImplicitImputation(d_input)[source]¶

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

class pypots.nn.modules.vader.GMMLayer(d_hidden, n_clusters)[source]¶

forward()[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor, Tensor]

Note

pypots.nn.modules.patchtst¶

The package including the modules of PatchTST.

Refer to the paper Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/yuqinie98/PatchTST

class pypots.nn.modules.patchtst.PatchtstEncoder(n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

forward(x, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.patchtst.PatchEmbedding(d_model, patch_size, patch_stride, padding, dropout, positional_embedding=True)[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.patchtst.RegressionHead(n_features, d_model, d_output, head_dropout, y_range=None)[source]¶

forward(x)[source]¶: x: [bs x nvars x d_model x num_patch] output: [bs x output_dim]

class pypots.nn.modules.patchtst.ClassificationHead(n_features, d_model, n_classes, head_dropout)[source]¶

forward(x)[source]¶: x: [bs x nvars x d_model x num_patch] output: [bs x n_classes]

class pypots.nn.modules.patchtst.PredictionHead(d_model, n_patches, n_steps_forecast, head_dropout=0, individual=False, n_features=0)[source]¶

forward(x)[source]¶: x: [bs x nvars x d_model x num_patch] output: [bs x forecast_len x nvars]

pypots.nn.modules.timemixer¶

The package including the modules of TimeMixer.

Refer to the paper Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou. “TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting”. In ICLR 2024.

Notes

This implementation is inspired by the official one https://github.com/kwuking/TimeMixer

class pypots.nn.modules.timemixer.BackboneTimeMixer(task_name, n_steps, n_features, n_pred_steps, n_pred_features, n_layers, d_model, d_ffn, dropout, top_k, channel_independence, decomp_method, moving_avg, downsampling_layers, downsampling_window, downsampling_method, use_future_temporal_feature, use_norm=False, embed='fixed', freq='h', n_classes=None)[source]¶

pypots.nn.modules.gpt4ts¶

The package including the modules of GPT4TS.

Refer to the paper Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, Rong Jin. One Fits All: Power General Time Series Analysis by Pretrained LM. NeurIPS 2023.

Notes

This implementation is inspired by the official one https://github.com/DAMO-DI-ML/NeurIPS2023-One-Fits-All

class pypots.nn.modules.gpt4ts.BackboneGPT4TS(task_name, n_steps, n_features, n_pred_steps, n_pred_features, n_layers, patch_size, patch_stride, train_gpt_mlp, d_ffn, dropout, embed, freq, n_classes=None)[source]¶

forward(x_enc, x_mark_enc=None, mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.moderntcn¶

The package including the modules of ModernTCN.

Refer to the paper Donghao Luo, and Xue Wang. ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis. In The Twelfth International Conference on Learning Representations. 2024.

Notes

This implementation is inspired by the official one https://github.com/luodhhh/ModernTCN

class pypots.nn.modules.moderntcn.BackboneModernTCN(n_steps, n_features, n_predict_features, patch_size, patch_stride, downsampling_ratio, ffn_ratio, num_blocks, large_size, small_size, dims, small_kernel_merged=False, backbone_dropout=0.1, head_dropout=0.1, use_multi_scale=True, individual=False, freq='h')[source]¶

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.crossformer¶

The package including the modules of Crossformer.

Refer to the paper Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The 11th ICLR, 2023.

Notes

This implementation is inspired by the official one https://github.com/Thinklab-SJTU/Crossformer

class pypots.nn.modules.crossformer.CrossformerEncoder(attn_layers)[source]¶

forward(x, src_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.CrossformerDecoder(layers)[source]¶

forward(x, cross)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.TwoStageAttentionLayer(seg_num, factor, d_model, n_heads, d_k, d_v, d_ff=None, dropout=0.1, attn_dropout=0.1)[source]¶

The Two Stage Attention (TSA) Layer input/output shape: [batch_size, Data_dim(D), Seg_num(L), d_model]

forward(x)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.ScaleBlock(win_size, d_model, n_heads, d_ff, depth, dropout, seg_num, factor)[source]¶

forward(x, attn_mask=None, tau=None, delta=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.crossformer.CrossformerDecoderLayer(self_attention, cross_attention, seg_len, d_model, d_ff=None, dropout=0.1)[source]¶

forward(x, cross)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.grud¶

The package including the modules of GRU-D.

Refer to the paper Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports, 8(1):6085, April 2018.

Notes

This implementation is inspired by the official one https://github.com/PeterChe1990/GRU-D

class pypots.nn.modules.grud.BackboneGRUD(n_steps, n_features, rnn_hidden_size)[source]¶

forward(X, missing_mask, deltas, empirical_mean, X_filledLOCF)[source]¶

Forward processing of GRU-D.

Parameters:

X
missing_mask
deltas
empirical_mean
X_filledLOCF

Return type:

Tuple[Tensor, ...]

Returns:

classification_pred
logits

class pypots.nn.modules.grud.TemporalDecay(input_size, output_size, diag=False)[source]¶

The module used to generate the temporal decay factor gamma in the GRU-D model. Please refer to the original paper [49] for more details.

Attributes:

W (tensor,) – The weights (parameters) of the module.
b (tensor,) – The bias of the module.

Parameters:

input_size (int,) – the feature dimension of the input
output_size (int,) – the feature dimension of the output
diag (bool,) – whether to product the weight with an identity matrix before forward processing

References

forward(delta)[source]¶

Forward processing of this NN module.

Parameters:: delta (tensor, shape [n_samples, n_steps, n_features]) – The time gaps.
Returns:: gamma – The temporal decay factor.
Return type:: tensor, of the same shape with parameter delta, values in (0,1]

pypots.nn.modules.saits¶

The package including the modules of SAITS.

Refer to the paper Wenjie Du, David Cote, and Yan Liu. SAITS: Self-Attention-based Imputation for Time Series. Expert Systems with Applications, 219:119619, 2023.

Notes

This implementation is inspired by the official one https://github.com/WenjieDu/SAITS

class pypots.nn.modules.saits.BackboneSAITS(n_steps, n_features, n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

forward(X, missing_mask, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.saits.SaitsEmbedding(d_in, d_out, with_pos, n_max_steps=1000, dropout=0)[source]¶

The embedding method from the SAITS paper [1].

Parameters:

d_in (int) – The input dimension.
d_out (int) – The output dimension.
with_pos (bool) – Whether to add positional encoding.
n_max_steps (int) – The maximum number of steps. It only works when with_pos is True.
dropout (float) – The dropout rate.

forward(X, missing_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.saits.SaitsLoss(ORT_weight, MIT_weight, loss_calc_func=MAE())[source]¶

forward(reconstruction, X_ori, missing_mask, indicating_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.csdi¶

The package including the modules of CSDI.

Refer to the paper Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. In NeurIPS, 2021.

Notes

This implementation is inspired by the official one the official implementation https://github.com/ermongroup/CSDI.

class pypots.nn.modules.csdi.BackboneCSDI(n_layers, n_heads, n_channels, d_target, d_time_embedding, d_feature_embedding, d_diffusion_embedding, is_unconditional, n_diffusion_steps, schedule, beta_start, beta_end)[source]¶

forward(observed_data, cond_mask, side_info, n_sampling_times)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csdi.CsdiDiffusionEmbedding(n_diffusion_steps, d_embedding=128, d_projection=None)[source]¶

forward(diffusion_step)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csdi.CsdiDiffusionModel(n_diffusion_steps, d_diffusion_embedding, d_input, d_side, n_channels, n_heads, n_layers)[source]¶

forward(x, cond_info, diffusion_step)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.csdi.CsdiResidualBlock(d_side, n_channels, diffusion_embedding_dim, nheads)[source]¶

forward(x, cond_info, diffusion_emb)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.timemixerpp¶

The package including the modules of TimeMixer++.

Refer to the paper Shiyu Wang, Jiawei Li, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Ju Shengtong, Zhixuan Chu, Ming Jin. “TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis”. ICLR 2025.

Notes

This implementation is inspired by the official one https://anonymous.4open.science/r/TimeMixerPP

class pypots.nn.modules.timemixerpp.BackboneTimeMixerPP(task_name, n_steps, n_features, n_pred_steps, n_pred_features, n_layers, d_model, d_ffn, n_heads, dropout, top_k, n_kernels, channel_mixing, channel_independence, downsampling_layers, downsampling_window, downsampling_method, use_future_temporal_feature, use_norm=False, embed='fixed', freq='h', n_classes=None)[source]¶

pypots.nn.modules.gpvae¶

The package including the modules of GP-VAE.

Refer to the paper Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pages 1651–1661. PMLR, 2020.

Notes

This implementation is inspired by the official one https://github.com/ratschlab/GP-VAE

class pypots.nn.modules.gpvae.BackboneGPVAE(input_dim, time_length, latent_dim, encoder_sizes=(64, 64), decoder_sizes=(64, 64), beta=1, M=1, K=1, kernel='cauchy', sigma=1.0, length_scale=7.0, kernel_scales=1, window_size=24)[source]¶

model GPVAE with Gaussian Process prior

Parameters:

input_dim (int,) – the feature dimension of the input
time_length (int,) – the length of each time series
latent_dim (int,) – the feature dimension of the latent embedding
encoder_sizes (tuple,) – the tuple of the network size in encoder
decoder_sizes (tuple,) – the tuple of the network size in decoder
beta (float,) – the weight of the KL divergence
M (int,) – the number of Monte Carlo samples for ELBO estimation
K (int,) – the number of importance weights for IWAE model
kernel (str,) – the Gaussian Process kernel [“cauchy”, “diffusion”, “rbf”, “matern”]
sigma (float,) – the scale parameter for a kernel function
length_scale (float,) – the length scale parameter for a kernel function
kernel_scales (int,) – the number of different length scales over latent space dimensions

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.fedformer¶

The package including the modules of FEDformer.

Refer to the paper Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 27268–27286. PMLR, 17–23 Jul 2022.

Notes

This implementation is inspired by the official one https://github.com/MAZiqing/FEDformer

class pypots.nn.modules.fedformer.FEDformerEncoder(n_steps, n_layers, d_model, n_heads, d_ffn, moving_avg_window_size, dropout, version='Fourier', modes=32, mode_select='random', activation='relu')[source]¶

forward(X, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.fedformer.FEDformerDecoder(n_steps, n_pred_steps, n_layers, n_heads, d_model, d_ffn, d_output, moving_avg_window_size, dropout, version='Fourier', modes=32, mode_select='random', activation='relu')[source]¶

forward(X, attn_mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

class pypots.nn.modules.fedformer.MultiWaveletTransform(ich=1, k=8, alpha=16, c=128, nCZ=1, L=0, base='legendre', attention_dropout=0.1)[source]¶

1D multiwavelet block.

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

class pypots.nn.modules.fedformer.MultiWaveletCross(in_channels, out_channels, seq_len_q, seq_len_kv, modes, c=64, k=8, ich=512, L=0, base='legendre', mode_select_method='random', initializer=None, activation='tanh', **kwargs)[source]¶

1D Multiwavelet Cross Attention layer.

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

class pypots.nn.modules.fedformer.FourierBlock(in_channels, out_channels, seq_len, modes=0, mode_select_method='random')[source]¶

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

class pypots.nn.modules.fedformer.FourierCrossAttention(in_channels, out_channels, seq_len_q, seq_len_kv, modes=64, mode_select_method='random', activation='tanh', policy=0, num_heads=8)[source]¶

forward(q, k, v, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, None]

Note

pypots.nn.modules.nonstationary_transformer¶

The package including the modules of Non-stationary Transformer.

Refer to the paper Yong Liu, Haixu Wu, Jianmin Wang, Mingsheng Long. Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting. Advances in Neural Information Processing Systems 35 (2022): 9881-9893.

Notes

This implementation is inspired by the official one https://github.com/thuml/Nonstationary_Transformers

class pypots.nn.modules.nonstationary_transformer.NonstationaryTransformerEncoder(n_layers, d_model, n_heads, d_k, d_v, d_ffn, dropout, attn_dropout)[source]¶

NonstationaryTransformer encoder. Its arch is the same with the original Transformer encoder, but the attention operator is replaced by the DeStationaryAttention.

Parameters:

n_layers (int) – The number of layers in the encoder.
d_model (int) – The dimension of the module manipulation space. The input tensor will be projected to a space with d_model dimensions.
n_heads (int) – The number of heads in multi-head attention.
d_k (int) – The dimension of the key and query tensor.
d_v (int) – The dimension of the value tensor.
d_ffn (int) – The dimension of the hidden layer in the feed-forward network.
dropout (float) – The dropout rate.
attn_dropout (float) – The dropout rate for the attention map.

forward(x, src_mask=None, **kwargs)[source]¶

Forward processing of the encoder.

Parameters:

x (Tensor) – Input tensor.
src_mask (Optional[Tensor]) – Masking tensor for the attention map. The shape should be [batch_size, n_heads, n_steps, n_steps].

Return type:

Union[Tensor, Tuple[Tensor, list]]

Returns:

enc_output – Output tensor.
attn_weights_collector – A list containing the attention map from each encoder layer.

class pypots.nn.modules.nonstationary_transformer.DeStationaryAttention(temperature, attn_dropout=0.1)[source]¶

De-stationary Attention

forward(q, v, k, attn_mask=None, **kwargs)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.nonstationary_transformer.Projector(d_in, n_steps, d_hidden, n_hidden_layers, d_output, kernel_size=3)[source]¶

MLP to learn the De-stationary factors

forward(x, stats)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.crli¶

The package including the modules of CRLI.

Refer to the paper Qianli Ma, Chuxin Chen, Sen Li, and Garrison W. Cottrell. Learning Representations for Incomplete Time Series Clustering. In AAAI, 35(10):8837–8846, May 2021.

Notes

This implementation is inspired by the official one https://github.com/qianlima-lab/CRLI

class pypots.nn.modules.crli.BackboneCRLI(n_steps, n_features, n_generator_layers, rnn_hidden_size, decoder_fcn_output_dims, rnn_cell_type='GRU')[source]¶

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note

class pypots.nn.modules.crli.CrliGenerator(n_layers, n_features, d_hidden, cell_type)[source]¶

forward(X, missing_mask)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.crli.CrliDecoder(n_steps, d_input, d_output, fcn_output_dims=None)[source]¶

forward(generator_fb_hidden_states)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, Tensor]

Note

class pypots.nn.modules.crli.CrliDiscriminator(cell_type, d_input)[source]¶

forward(X, missing_mask, imputation_latent)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tensor

Note

pypots.nn.modules.ts2vec¶

The package including the modules of TS2Vec.

Refer to the paper Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, Bixiong Xu. “TS2Vec: Towards Universal Representation of Time Series”. In AAAI 2022.

Notes

This implementation is inspired by the official one https://github.com/zhihanyue/ts2vec

class pypots.nn.modules.ts2vec.TS2VecEncoder(n_features, n_pred_features, d_hidden, n_layers, mask_mode='binomial')[source]¶

forward(x, mask=None)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

encode(x, mask=None, encoding_window=None, causal=False, sliding_length=None, sliding_padding=0)[source]¶

Compute representations using the trained model.

Parameters:

x (Tensor) – This should have a shape of (n_samples, n_steps, n_features). All missing data should be set to NaN.
mask (Optional[str]) – The mask used by encoder can be specified with this parameter. This can be set to ‘binomial’, ‘continuous’, ‘all_true’, ‘all_false’ or ‘mask_last’.
encoding_window (Optional[str]) – When this param is specified, the computed representation would the max pooling over this window. This can be set to ‘full_series’, ‘multiscale’ or an integer specifying the pooling kernel size.
causal (bool) – When this param is set to True, the future information would not be encoded into representation of each timestamp.
sliding_length (Optional[int]) – The length of sliding window. When this param is specified, a sliding inference would be applied on the time series.
sliding_padding (int) – This param specifies the contextual data length used for inference every sliding windows.

Return type:

Tensor

Returns:

repr: The representations for data.

pypots.nn.modules.totem¶

The package including the modules of TOTEM.

Refer to the paper Sabera J Talukder, Yisong Yue, and Georgia Gkioxari. TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis. In TMLR, 2024.

Notes

This implementation is inspired by the official one https://github.com/SaberaTalukder/TOTEM

class pypots.nn.modules.totem.VQVAE(block_hidden_size, num_residual_layers, res_hidden_size, embedding_dim, num_embeddings, commitment_cost, compression_factor)[source]¶

forward(X)[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

pypots.nn.modules.usgan¶

The package including the modules of USGAN.

Refer to the paper Xiaoye Miao, Yangyang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative Semi-supervised Learning for Multivariate Time Series Imputation. In AAAI, 35(10):8983–8991, May 2021.

class pypots.nn.modules.usgan.BackboneUSGAN(n_steps, n_features, rnn_hidden_size, lambda_mse, hint_rate=0.7, dropout_rate=0.0)[source]¶

USGAN model

forward(inputs, training_object='generator')[source]¶

Defines the computation performed at every call.

Should be overridden by all subclasses. :rtype: Tuple[Tensor, ...]

Note