Introduction

Cryptocurrency is a form of digital currency that regulates the generation of currency units and
verifies the transfer of funds using encryption techniques. Notably, cryptocurrencies are not
governed by a central authority and operate on a decentralized structure. Since the launch of
Bitcoin in 2009, cryptocurrencies have revolutionized the way people transfer money.
Cryptocurrency was first proposed in 1998 by a computer scientist, Wei Dai, who developed a
cryptography-based system that could be used to ease payments between parties. This system,
called "b-money," laid the groundwork for future cryptocurrencies.

The systematic structural specification of Bitcoin [1] was published in November 2008 by an
unknown individual or group using the alias Satoshi Nakamoto. Bitcoin was the first
cryptocurrency to be decentralized. Since the introduction of Bitcoin in 2009, cryptocurrencies
have transformed how money is sent and received. Bitcoin is still the most popular and valuable
cryptocurrency in the world, despite the creation of thousands of other cryptocurrencies and
several price fluctuations since then. At the time of this writing, Bitcoin's market capitalization
exceeds 475 billion US dollars. In addition, the market capitalization of all active
cryptocurrencies, including Bitcoin, reaches 1.17 trillion US dollars. [2]

Due to the decentralized nature of the majority of cryptocurrencies, their prices are not
influenced by interest rates, inflation rates, or monetary policies, but rather by the perception of
users based on news, websites, and other non-fundamental elements. [3] The stock markets
are influenced by a variety of factors that create uncertainty, including political and economic
issues that have a local or global impact. Understanding the success keys, or factors that
provide accurate predictions, is a difficult task. We can examine the market using any technique,
including technical indicators, price fluctuations, and market technical analysis. There is thus a
need for automated prediction tools to assist investors in deciding whether to invest in bitcoin or
other cryptocurrencies. Modern stock market predictions typically include automation
technologies, and we could apply the same approach and strategy to this realm of
cryptocurrency.

Random Forest

Random Forest is a machine learning technique that employs decision trees to define a
prediction model [4]. Several decision trees are utilized by the Random Forest technique to
forecast Bitcoin prices [5]. Random forests have been used to forecast the direction of a stock
price trend. In 2016, Khaidem et al. employed random forests to forecast trends in the values of
Apple, Samsung Electronics, and General Electric shares traded on the NASDAQ.

Specifically, Random Forest is an ensemble technique consisting of decision trees and bagging
[6]. It causes different data samples to be used to train each tree for the same problem. Different
trees view distinct chunks of the data while employing bagging. No tree is exposed to all training
data. So, when integrating their results, certain inaccuracies are compensated for others,
resulting in a more generalizable prediction [7].

Overfitting is a major issue that can have devastating effects on results; however, if there are
enough trees in the forest, the Random Forest algorithm will not overfit the model. Moreover,
missing values can be handled by the random forest. As previously stated, the random forest is
based on a different technique known as bagging, which entails running a bootstrap on the
experiment's data sampling on the input variables and employing a fixed number of variables.

The random forest model takes into account the following:

A random sample of N cases from the training set is taken with replacement. This sample
constitutes the training set to build tree i. Given that the number of input variables is M, the
number of variables selected for each node is m (m<M), which remains constant during block
generation. The block is then associated with the node by splitting it and utilizing the optimal
division of its m properties. The number of predictors evaluated in each division is roughly equal
to the square root of the total number of predictors. The most effective method for determining
the appropriate value is to analyze the out-of-bag MSE for various values of m. In general, if the
variables chosen at each node are highly connected, small values of m result in favorable
outcomes. As a result, at each node, m observations are used for training and M-m
observations are used for testing. Globally, given a training set D of size n, m new training sets
D(i = 1, . .., m) of size n′ are created from m samples with replacement. Afterwards, each
decision tree is trained using the data set D. In addition, it enters the nodes of each tree in order
to make a forecast when it has a new observation, as shown in figure.

Source: [Link]

LSTM:

LSTM (Long Short Term Memory) is another form of RNN module. Hochreiter and Schmidhuber
(1997) [9] created LSTM, which was later developed and popularized by several researchers.
The LSTM network consists of modules with recurrent consistency, similar to an RNN. The
distinction between LSTM and RNN is the connectivity between the hidden layers of RNN. The
RNN explanation structure is depicted in Figure. The only distinction between RNN and LSTM is
the memory cell of the structure's hidden layer. And the design of three unique gates efficiently
resolves gradient issues. Figure depicts the LSTM memory structure of the hidden layer. [10]

The Expanded Structure of RNN

M. W. P. Aldi, J. Jondri, and A. Aditsania, “Analisis Dan Implementasi Long Short Term Memory
Neural Network Untuk Prediksi Harga Bitcoin,” eProceedings Eng., vol. 5, no. 2, 2018.

Figure explains that the RNN has deficiencies, which may be observed in the input. This
problem was discovered by Bengio, et al. (1994)[11]. 𝑋0, 𝑋1 have a very large range of
information 𝑋𝑡, 𝑋𝑡+1, so that when ℎ𝑡+1 requires information, those that are relevant to 𝑋0, 𝑋1 in
the RNN are unable to learn to link information. Because the old memory that is saved becomes
increasingly useless over time given that it is overwritten or replaced by new memory.

Source: [Link]

The LSTM's special units(recurrent hidden layers) contain memory blocks. In addition to
memory cells with self-connections that store the network's temporal state, the memory blocks
also contain multiplicative units called gates that regulate the flow of information. In the original
architecture, each memory block comprised an input gate and an output gate. Controlling the
flow of input activations into the memory cell is the input gate. The output gate regulates the
flow of cell activations from the cell to the remainder of the network. Subsequently, the memory
block received the forget gate [12]. This addressed a shortcoming of LSTM models that
prevented them from processing continuous input streams that were not divided into
subsequences. The forget gate scales the internal state of the cell prior to adding it as input to
the cell via its self-recurrent link, thereby forgetting or resetting the cell's memory in an adaptive
manner. In addition, the contemporary LSTM architecture includes peephole connections from
its internal cells to the gates in the same cell in order to learn precise output timing [13].

A mapping from an input sequence x = (𝑋0, 𝑋1, ..., 𝑋𝑡) to an output sequence x = (𝑌0, 𝑌1, ..., 𝑌𝑡)
is computed by an LSTM network through calculating the network unit activations iteratively
from i=1 to t as follows:

In these formulas, σ, i, f, o and c are respectively the logistic sigmoid function, input gate, forget
gate, output gate, and cell activation vectors, all of which are the same size as the cell output
activation vector m. W denote as weight matrices (e.g. 𝑊𝑖𝑥 is the matrix of weights from the
input gate to the input, 𝑊𝑖𝑐, 𝑊𝑓𝑐, 𝑊𝑜𝑐 are diagonal weight matrices for peephole connections),
the b terms denote bias vectors (e.g. 𝑏𝑖 is the input gate bias vector). Furthermore, g and h are
the cell input and cell output activation functions, respectively, and tanh and are the network
output activation functions, respectively, and softmax is present.

GRU:

A gated recurrent unit (GRU) was presented [14] to enable each recurrent unit to capture
adaptive dependencies on several time scales. Similar to the LSTM unit, the GRU possesses
gating units that influence the flow of information inside the unit, but without distinct memory
cells.

The architecture of the Gated Recurrent Unit:

Here we have a GRU cell that is comparable to an LSTM cell or RNN cell.
At each timestamp t, it receives an input 𝑋𝑡 and the preceding timestamp's hidden state 𝐻𝑡−1.
Subsequently, it outputs a new hidden state 𝐻𝑡, which is consequently sent to the subsequent
timestamp. Currently, a GRU cell consists mostly of two gates, as opposed to three gates in an
LSTM cell. The initial gate is the Reset gate, while the second is the update gate.

Reset Gate (short-term memory):

The reset gate is responsible for the short-term memory of the network i.e the hidden state (𝐻𝑡).
Here is the equation of the reset gate.

It resembles the LSTM gate equation. The sigmoid function limits 𝑟𝑡 to (0-1). reset gate weight
matrices 𝑈𝑟 and 𝑊𝑟.

Update Gate (long-term memory):

Similarly, we have an update gate for long-term memory and the equation of the gate is shown
below.

The only difference is in the weight metrics, i.e., 𝑈𝑢 and 𝑊𝑢.

Current Memory Gate:

During a normal explanation of Gated Recurrent Unit Network, it is frequently disregarded. It is a
component of the reset gate, just as the Input Modulation Gate is a subcomponent of the Input
Gate, and is used to introduce nonlinearity to the input and make it zero-mean. A second reason
to include it as a sub-part of the Reset Gate is to lessen the impact that past information has on
the information that is being transmitted into the future. The calculation is as follows:

Orbit

Uber's Orbit is an open source package designed to ease time series inferences and forecasts
using structural Bayesian time series models for real-world applications and scientific study
[16].It employs probabilistic programming languages such as Stan [17] and Pyro [18] while
providing a familiar and intuitive initialize-fit-predict interface for time series workloads.

It introduces a collection of refined Bayesian exponential smoothing models with a wide range of
priors, model type specifications, and noise distribution options. The model includes a novel
global trend term that is effective for short-term time series. Most significantly, it includes a
well-crafted Python compute software/package named Orbit (Object-oriented Bayesian Time
Series). The underlying MCMC sampling process and optimization are handled using the
probabilistic programming languages Stan (Carpenter et al., 2017) and Pryo (Bingham et al.,
2019). Pyro, created by Uber researchers, is a universal probabilistic programming language
(PPL) built in Python and backed on the backend by PyTorch and JAX. Orbit presently has a
subset of the available prediction and sampling algorithms for Pyro estimating.

Arima

The Autoregressive Integrated Moving Average (ARIMA) method was developed in 1970 by
George Box and Gwilyn Jenkins and is also known as the BoxJenskins method [19]. The
ARIMA method completely ignores independent variables while predicting, making it suited for
interconnected statistical data (dependent) and requiring some assumptions such as
autocorrelation, trend, or seasonality. The ARIMA method can predict historical data with the
influence of difficult-to-understand data, has a high degree of accuracy in short-term forecasting,
and can deal with seasonal data variations.

The ARIMA method is classified into four categories: Autoregressive (AR), Moving Average
(MA), Autoregressive Moving Average (ARMA), and Autoregressive Integrated Moving Average
(ARIMA) [20][21].

1- Autoregressive (AR):
It was introduced by Yule in 1926 and expanded by Walker in 1932. This model assumes that
data from prior periods is currently influencing current data. It's called autoregressive since it's
rebuilt against the variable's prior values in this model. The AR method is used to calculate the
order value of the coefficient p, which represents a value's dependence on its previous nearest
value [22].

The the general form of an AR model with order p (AR (p)) or an ARIMA model (𝑝, 0, 0) is as
follows:

2- Moving Average (MA):

It was first presented by Slutzky in 1973. The MA approach is used to calculate the order
coefficient q, which accounts for the varied movement of the prior residual value [18]. The
following is the generic form of the MA model with order 𝑞 (𝑀𝐴(𝑞) ) or ARIMA model (0, 0, 𝑞) .

3- Autoregressive Moving Average (ARMA):

This model combines the AR and MA models. Assume that the current period data is influenced
by prior period data and the preceding period's forced value. The following are common forms of
the AR and MA or ARIMA processes' models (𝑝, 0, 𝑞) .

4- Autoregressive Integrated Moving Average (ARIMA):

The ARIMA Model assumes that the data used must be stationary, which means that the
average variation of the data is constant. Nonstationary data must first be transformed into
stationary data using a differencing procedure. The ARIMA technique is a statistical viewpoint
method represented by three parameters, the first of which is the data AR process of the
previous period, which is taken and maintained later in the Integrated process making the data
easier to predict. The common version of the ARIMA model (𝑝, 𝑑, 𝑞) is as follows.

Sarimax

ARMA is the combination of the AR and MA models. Adding an integration operator to an

ARMA model produces an ARIMA model. A SARIMAX model incorporates exogenous variables
assessed at time t that influence the value of input data at time t and integer multipliers of
seasonality [21]. The parameters required to define the SARIMAX model are listed in Table1.

Source: Studies on Day-ahead Electricity Price Forecasting: SARIMAX, LSTM and CNN-LSTM
with the Electrical Consumption as an Exogenous Input

The formula for the Autoregressive Model (AR(p)) is:

The Moving Average Model (MA(d)) is represented as follows:

Autoregressive Moving Average Model (ARMA(p,q)) can be written as:

and the Autoregressive Integrated Moving Average Model (ARIMA(p,d,q)) is stated as:

Finally, Seasonal Autoregressive Integrated Moving Average Model with Explicative Variable
(SARIMAX((p, d, q) ∗ (P, D, Q)) is stated as follows:

Prophet

Prophet is a method for forecasting time series data based on an additive model in which
non-linear trends are fitted with annual, weekly, and daily seasonality, in addition to holiday
effects. It is most effective when applied to time series with substantial seasonal effects and
multiple seasons of historical data. Prophet is robust to missing data and fluctuations in the
trend, and it typically handles outliers well [23].
As long as the Prophet accurately displays the conditional mean and conditional variance, it
should function adequately. Mathematically, we have this formula:

This could be the case if the underlying system is in a condition of equilibrium, such as when the
economy is stable. Therefore, once a significant shock occurs, the variance criterion will very
certainly be violated. This is precisely what we observed in the preceding time series example.

XGBOOST
Gradient boosting (GBM) trees do unsupervised learning, in which they learn from data without
a specified model. XGBoost is a popular gradient-boosting library. It can be used for GPU
training, distributed computing, and parallelization. It is accurate, adaptable to all forms of data
and situations, well-documented, and extremely user-friendly.

XGBoost is an abbreviation for Extreme Gradient Boosting. It is a properly parallelized and

optimized version of the gradient boosting technique. Parallelizing the entire boosting procedure
drastically reduces training time.

Rather than training the best possible model on the data (as is the case with conventional
approaches), they trained hundreds of models on various subsets of the training dataset and
then conducted a vote to determine the model with the best performance.

In many situations, XGBoost is superior to conventional gradient-boosting methods. The Python

implementation provides access to a huge array of inner parameters that can be modified to
improve precision and accuracy.

Parallelization, regularization, non-linearity, cross-validation, and scalability are some of

XGBoost's most essential characteristics.

The XGBOOST algorithm works in such a way that it considers or estimates a function. To
begin, we generate a sequence based on the function gradients. The following equation models
a specific type of gradient descent. It specifies the direction in which the function decreases, as
it represents the loss function to minimize. corresponds to the learning rate in gradient descent
and is the rate of change fitted to the loss function. is anticipated to replicate the loss's behavior
adequately.

To iterate over the model and determine its optimal formulation, we must describe the entire
formula as a sequence and identify a function that will converge to the function's minimum. This
function will serve as an error metric to help us minimize losses and sustain performance over
time. The series approaches the minimal value of the function. This specific notation denotes
the error function that applies while evaluating a gradient boosting regressor [25].

Model Performance:

Cross Validation:

Cross-validation (CV) is a well-known method for adjusting hyperparameters and generating

accurate measures of model performance. k-fold cross-validation and hold-out cross-validation
are two of the most prevalent types of cross-validation.

First, we divided the dataset into the training set and the test set. If any parameters require
tuning, we divide the training set into a training subset and a validation subset. On the training
subset, the model is trained, and the parameters that minimize error on the validation set are
selected. Finally, the model is trained on the complete training set using the selected
parameters, and the error on the test set is recorded.

Source: [Link]
But there is a serious difference when we want to use cross validation in time series. Because of
two reasons:

1. Temporal Dependencies:

Particular care must be used while separating time series data in order to prevent data
leaking. To accurately imitate the "actual world forecasting environment, in which we
stand in the present and predict the future"[27] , the forecaster must withhold any data
regarding events that occur after the events used for fitting the model. Instead of k-fold
cross-validation, we employ hold-out cross-validation for time series data, where a
portion of the data (temporally separated) is reserved for assessing the model's
performance. For example, observe the Split 4 in the above Figure where the validation
data which is Fold 4 comes before the Fold 5 which is the part of the training data.

2. Arbitrary Choice of Test Set:

You may notice that the selection of the test set in Hold-out validation is somewhat
random, which may imply that our test set error is an inadequate estimate of error on a
separate test set. To overcome this, we employ the Nested Cross-Validation approach.
Nested CV has an outside loop for error estimates and an inner loop for parameter
adjustment (see the Figure below). The inner loop operates precisely as previously
described: the training set is divided into a training subset and a validation set, the model
 is trained on the training subset, and the parameters that minimize error on the validation
set are selected. Nevertheless, we now include an outside loop that divides the dataset
into numerous training and test sets, and the error on each split is averaged to obtain a
robust estimate of model error. This is beneficial because:

Source:
[Link]

Method:

After setting the number of splits, the training dataset is splitted to equal subsets. For instance,
𝑡ℎ
there are 6 equal subsets in the above figure. In the 𝑘 iteration of the outer loop,the k first
𝑡ℎ
subsets are considered as a training subset and 𝑘 + 1 is considered as a validation subset.

Metrics:

After the obtaining the final predictions of the model, validating the data will usually be carried
out by calculating[28]:

Mean Absolute Error (MAE):

It is the mean of the absolute value of the differences between the forecasting price and the
actual value. It is easy to interpret and it benefits you by offering errors in the units of the data
and the prediction. However, it does not penalize outliers(which could be not very important in
price prediction). The most notable drawback of this metric is that it is scale dependent so we
can not compare different cryptocurrencies with different units.

Mean Squared Error (MSE):

It is the mean of the square of the differences between the forecasting price and the actual
value. In this metric, outliers are heavily punished. On the other hand, as the error is not in the
original units of the data and prediction, it is harder to interpret. It is scale dependent as well so
we have the similar issue like in MAE.

Root Mean Squared Error (RMSE):

It is the same as MSE apart from at the end we square root the [Link] this metric, outliers are
heavily punished like MSE and it has the strong point of being in the units of the data and
prediction. It is kind of the best of both MSE and MAE worlds. But, since you square the error, it
could be still less interpretable. Furthermore, it is scale dependent.

Mean Absolute Percentage Error (MAPE):

Mean absolute percentage error is the mean of the percentage difference between the actual
value and prediction. This is often used as the baseline metric to measure most forecasting
models. MAPE is not only easy to interpret but also scale independent which allows us to
compare different cryptocurrencies. Nevertheless, in the cryptocurrencies with actual value near
zero, we could have infinite error. In this metric, lower forecasts are bound to 100% error but
higher forecasts could increase to infinite error, thus, it is biased to under-forecast.

Symmetric Mean Absolute Percentage Error (SMAPE):

It is an extension of MAPE. It is the mean of the 200 times the difference between the actual
value and prediction divided by the sum of their absolute values. It no longer favors the under
forcastings. It is now fully bounded between 0% and 200%. Since the denominator could be still
around zero, there is still a chance of infinite values. Additionally, interpreting a metric between
0% and 200% could be difficult. One supposed problem with SMAPE is that it is not symmetric
since over and under forecasts are not treated equally. This is illustrated by the following example by
applying the SMAPE formula:
● Over-forecasting: At = 100 and Ft = 110 give SMAPE = 9.09%
● Under-forecasting: At = 100 and Ft = 90 give SMAPE = 10.52%
Mean Absolute Scaled Error (MASE):
It has two modes.
In case of no seasonality:

In case of seasonality:

This is the mean absolute scaled error for both seasonal and non-seasonal time series and is
probably the best and most fair metric to use. This metric compares the output to the naive forecast.

Naive forecasts are the most cost-effective forecasting model, and provide a benchmark against
which more sophisticated models can be compared. This forecasting method is only suitable for time
series data. Using the naive approach, forecasts are produced that are equal to the last observed
value. This method works quite well for economic and financial time series, which often have
patterns that are difficult to reliably and accurately predict. If the time series is believed to have
seasonality, the seasonal naive approach may be more appropriate where the forecasts are equal to
the value from last season. In time series notation:

In MASE, if the error is less than one, then it can be concluded that the forecast is better than an
averaged naive forecast. On the contrary, if there is more than one, the forecast is worse than an
averaged naive forecast.
The advantages of this metric are scale independency and penalizing under and over forecasting
equally.

Mean Squared Logarithmic Error (MSLE):

It measures the ratio or relative difference between the actuals and the forecasted values by
calculating the mean of the square of the logarithm of the actual value plus one divided by the
forecasted value plus one. This metric punishes the under forecasting more than over forecasting.
Nonetheless, it is not easy to interpret.

References
[1] S. Nakamoto, "Bitcoin: A Peer-to-Peer Electronic Cash System," 2008.
[2] "Coin Market Cap," [Online]. Available: [Link]
[Accessed 09 02 2023].
[3] Makarov, I., & Schoar, A. (2020). Trading and arbitrage in cryptocurrency markets. Journal of
Financial Economics, 135(2), 293–319
[4] Mitchell, T. M. (1999). Machine learning and data mining. Communications of the ACM,
42(11), 30–36
[5] Breiman, L. (2001). Random forests. Machine learning, 45(1), 5–32
[6] Breiman, L. (1996). Bagging predictors. Mach. Learn. 24, 123–140
[7] Biau, G. (2012). Analysis of a random forests model. The Journal of Machine Learning
Research, 13(1), 1063–1095
[8] A. Radityo, Q. Munajat, and I. Budi, “Prediction of Bitcoin exchange rate to American dollar
using artificial neural network methods,” in Advanced Computer Science and Information
Systems (ICACSIS), 2017 International Conference on, 2017, pp. 433–438
[9] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8,
pp. 1735– 1780, 1997.
[10] F. Qian and X. Chen, “Stock Prediction Based on LSTM under Different Stability,” in 2019
IEEE 4th International Conference on Cloud Computing and Big Data Analysis (ICCCBDA),
2019, pp. 483–486.
[11] Y. Bengio, P. Simard, and P. Frasconi, “Learning Long-Term Dependencies with Gradient
Descent is Difficult,” IEEE Trans. Neural Networks, 1994
[12] E. Kristensen, S. Østergaard, M. A. Krogh, and C. Enevoldsen, “Technical Indicators of
Financial Performance in the Dairy Herd,” J. Dairy Sci., 2008.
[13] C. Scheier and W. Tschacher, “Appropriate algorithms for nonlinear time series analysis in
psychology,” in Nonlinear dynamics in human behavior, World Scientific, 1996, pp. 27–43.
[14] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio “Empirical Evaluation
of Gated Recurrent Neural Networks on Sequence Modeling”
[15] [Link]
[16] Edwin Ng, Zhishi Wang, Huigang Chen, Steve Yang, and Slawek Smyl. 2021. Orbit:
Probabilistic Forecast with Exponential Smoothing. arXiv:2004.08492 [[Link]]
[17] Bob Carpenter, Andrew Gelman, Matthew D. Hoffman, Daniel Lee. “Stan : A Probabilistic
Programming Language”
[18] Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan,
Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, Noah D. Goodman. “Pyro: Deep
Universal Probabilistic Programming”
[19] E. B. Dagum, “The X-II-ARIMA seasonal adjustment method,” 2005.
[20] A. Hendranata, “ARIMA (Autoregressive Integrated Moving Average),” 2003.
[21] Y. S. Lee and L. I. Tong, “Forecasting time series using a methodology based on
autoregressive integrated moving average and genetic programming,” Knowledge-Based Syst.,
vol. 24, no. 1, pp. 66–72, Feb. 2011.
[22] Hillmer, S. Craig, and George C. Tiao, “An ARIMA-Model-Based Approach to Seasonal
Adjustment,” vol. 10, no. 1, pp. 5–24, 2017.
[23][Link]
[24] Sean J. Taylo, Benjamin Letham, “Forecasting at Scale”
[25]Tianqi Chen, Carlos Guestrin, “XGBoost: A Scalable Tree Boosting System”
[27] L. J. Tashman. Out-of-sample tests of forecasting accuracy: an analysis and review.
International Journal of Forecasting, 16(4):437–450, 2000.
[28][Link]
134