The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
Para muitos campos da vida real, a previsão de séries temporais é essencial. Estudos recentes mostraram que o Transformer tem certas vantagens ao lidar com tais problemas, especialmente ao lidar com problemas de entrada de tempo de sequência longa e de previsão de tempo de sequência longa. A fim de melhorar a eficiência e estabilidade local do Transformer, estes estudos combinam Transformer e CNN com diferentes estruturas. No entanto, os modelos de rede de previsão de séries temporais anteriores baseados no Transformer não podem fazer uso completo da CNN e não foram usados em uma melhor combinação de ambos. Em resposta a este problema na previsão de séries temporais, propomos o algoritmo de previsão de séries temporais baseado no Transformador de convolução. (1) Mecanismo de atenção ES: Combine a atenção externa com o mecanismo tradicional de autoatenção através da rede de dois ramos, o custo computacional do mecanismo de autoatenção é reduzido e a maior precisão de previsão é obtida. (2) Bloco de frequência aprimorada: Um bloco de frequência aprimorada é adicionado na frente do módulo ESAttention, que pode capturar estruturas importantes em séries temporais por meio do mapeamento no domínio de frequência. (3) Convolução dilatada causal: O módulo do mecanismo de autoatenção é conectado substituindo a camada de convolução padrão tradicional por uma camada de convolução dilatada causal, de modo que obtenha o campo receptivo de crescimento exponencial sem aumentar o consumo de cálculo. (4) Fusão de recursos multicamadas: As saídas de diferentes módulos do mecanismo de autoatenção são extraídas e as camadas convolucionais são usadas para ajustar o tamanho do mapa de recursos para a fusão. As informações de recursos mais refinadas são obtidas com um custo computacional insignificante. Experimentos em conjuntos de dados do mundo real mostram que a estrutura do modelo de previsão de rede de série temporal proposta neste artigo pode melhorar muito o desempenho da previsão em tempo real do atual modelo Transformer de última geração, e os custos de cálculo e memória são significativamente mais baixos. Comparado com algoritmos anteriores, o algoritmo proposto alcançou uma maior melhoria de desempenho tanto na eficácia quanto na precisão da previsão.
Na WANG
Nanjing University of Aeronautics and Astronautics,Nanjing Audit University Jinshen College
Xianglian ZHAO
Nanjing University of Aeronautics and Astronautics
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Copiar
Na WANG, Xianglian ZHAO, "Time Series Forecasting Based on Convolution Transformer" in IEICE TRANSACTIONS on Information,
vol. E106-D, no. 5, pp. 976-985, May 2023, doi: 10.1587/transinf.2022EDP7136.
Abstract: For many fields in real life, time series forecasting is essential. Recent studies have shown that Transformer has certain advantages when dealing with such problems, especially when dealing with long sequence time input and long sequence time forecasting problems. In order to improve the efficiency and local stability of Transformer, these studies combine Transformer and CNN with different structures. However, previous time series forecasting network models based on Transformer cannot make full use of CNN, and they have not been used in a better combination of both. In response to this problem in time series forecasting, we propose the time series forecasting algorithm based on convolution Transformer. (1) ES attention mechanism: Combine external attention with traditional self-attention mechanism through the two-branch network, the computational cost of self-attention mechanism is reduced, and the higher forecasting accuracy is obtained. (2) Frequency enhanced block: A Frequency Enhanced Block is added in front of the ESAttention module, which can capture important structures in time series through frequency domain mapping. (3) Causal dilated convolution: The self-attention mechanism module is connected by replacing the traditional standard convolution layer with a causal dilated convolution layer, so that it obtains the receptive field of exponentially growth without increasing the calculation consumption. (4) Multi-layer feature fusion: The outputs of different self-attention mechanism modules are extracted, and the convolutional layers are used to adjust the size of the feature map for the fusion. The more fine-grained feature information is obtained at negligible computational cost. Experiments on real world datasets show that the time series network forecasting model structure proposed in this paper can greatly improve the real-time forecasting performance of the current state-of-the-art Transformer model, and the calculation and memory costs are significantly lower. Compared with previous algorithms, the proposed algorithm has achieved a greater performance improvement in both effectiveness and forecasting accuracy.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2022EDP7136/_p
Copiar
@ARTICLE{e106-d_5_976,
author={Na WANG, Xianglian ZHAO, },
journal={IEICE TRANSACTIONS on Information},
title={Time Series Forecasting Based on Convolution Transformer},
year={2023},
volume={E106-D},
number={5},
pages={976-985},
abstract={For many fields in real life, time series forecasting is essential. Recent studies have shown that Transformer has certain advantages when dealing with such problems, especially when dealing with long sequence time input and long sequence time forecasting problems. In order to improve the efficiency and local stability of Transformer, these studies combine Transformer and CNN with different structures. However, previous time series forecasting network models based on Transformer cannot make full use of CNN, and they have not been used in a better combination of both. In response to this problem in time series forecasting, we propose the time series forecasting algorithm based on convolution Transformer. (1) ES attention mechanism: Combine external attention with traditional self-attention mechanism through the two-branch network, the computational cost of self-attention mechanism is reduced, and the higher forecasting accuracy is obtained. (2) Frequency enhanced block: A Frequency Enhanced Block is added in front of the ESAttention module, which can capture important structures in time series through frequency domain mapping. (3) Causal dilated convolution: The self-attention mechanism module is connected by replacing the traditional standard convolution layer with a causal dilated convolution layer, so that it obtains the receptive field of exponentially growth without increasing the calculation consumption. (4) Multi-layer feature fusion: The outputs of different self-attention mechanism modules are extracted, and the convolutional layers are used to adjust the size of the feature map for the fusion. The more fine-grained feature information is obtained at negligible computational cost. Experiments on real world datasets show that the time series network forecasting model structure proposed in this paper can greatly improve the real-time forecasting performance of the current state-of-the-art Transformer model, and the calculation and memory costs are significantly lower. Compared with previous algorithms, the proposed algorithm has achieved a greater performance improvement in both effectiveness and forecasting accuracy.},
keywords={},
doi={10.1587/transinf.2022EDP7136},
ISSN={1745-1361},
month={May},}
Copiar
TY - JOUR
TI - Time Series Forecasting Based on Convolution Transformer
T2 - IEICE TRANSACTIONS on Information
SP - 976
EP - 985
AU - Na WANG
AU - Xianglian ZHAO
PY - 2023
DO - 10.1587/transinf.2022EDP7136
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E106-D
IS - 5
JA - IEICE TRANSACTIONS on Information
Y1 - May 2023
AB - For many fields in real life, time series forecasting is essential. Recent studies have shown that Transformer has certain advantages when dealing with such problems, especially when dealing with long sequence time input and long sequence time forecasting problems. In order to improve the efficiency and local stability of Transformer, these studies combine Transformer and CNN with different structures. However, previous time series forecasting network models based on Transformer cannot make full use of CNN, and they have not been used in a better combination of both. In response to this problem in time series forecasting, we propose the time series forecasting algorithm based on convolution Transformer. (1) ES attention mechanism: Combine external attention with traditional self-attention mechanism through the two-branch network, the computational cost of self-attention mechanism is reduced, and the higher forecasting accuracy is obtained. (2) Frequency enhanced block: A Frequency Enhanced Block is added in front of the ESAttention module, which can capture important structures in time series through frequency domain mapping. (3) Causal dilated convolution: The self-attention mechanism module is connected by replacing the traditional standard convolution layer with a causal dilated convolution layer, so that it obtains the receptive field of exponentially growth without increasing the calculation consumption. (4) Multi-layer feature fusion: The outputs of different self-attention mechanism modules are extracted, and the convolutional layers are used to adjust the size of the feature map for the fusion. The more fine-grained feature information is obtained at negligible computational cost. Experiments on real world datasets show that the time series network forecasting model structure proposed in this paper can greatly improve the real-time forecasting performance of the current state-of-the-art Transformer model, and the calculation and memory costs are significantly lower. Compared with previous algorithms, the proposed algorithm has achieved a greater performance improvement in both effectiveness and forecasting accuracy.
ER -