Detecting PowerShell-based Fileless Cryptojacking

Attacks Using Machine Learning

Said Varlioglua,∗, Nelly Elsayeda , Murat Ozera , Zag ElSayeda , John M.
Emmerta,b
a
School of Information Technology, University of Cincinnati, Cincinnati, OH, USA
arXiv:2602.18285v1 [[Link]] 20 Feb 2026

b
Department of Electrical and Computer Engineering, University of
Cincinnati, Cincinnati, OH, USA

Abstract
With the emergence of remote code execution (RCE) vulnerabilities in ubiq-
uitous libraries and advanced social engineering techniques, threat actors
have started conducting widespread fileless cryptojacking attacks. These
attacks have become effective with stealthy techniques based on PowerShell-
based exploitation in Windows OS environments. Even if attacks are detected
and malicious scripts removed, processes may remain operational on victim
endpoints, creating a significant challenge for detection mechanisms. In this
paper, we conducted an experimental study with a collected dataset on de-
tecting PowerShell-based fileless cryptojacking scripts. The results showed
that Abstract Syntax Tree (AST)-based fine-tuned CodeBERT achieved a
high recall rate, proving the importance of the use of AST integration and
fine-tuned pre-trained models for programming language.
Keywords: Fileless Malware, Cryptojacking, PowerShell, Machine
Learning, Cybersecurity

∗
Corresponding author
Email addresses: varlioms@[Link] (John M. Emmert),
elsayeny@[Link] (John M. Emmert), ozermm@[Link] (John M.
Emmert), elsayezs@[Link] (John M. Emmert), emmertj@[Link]
(John M. Emmert)
1. Introduction
Cryptojacking, an unauthorized cryptocurrency mining on infected de-
vices, is a popular cyber attack based on financial motivations [1, 2]. De-
tection systems face challenges because of fileless attack techniques. Fileless
attacks have become popular with the prevalence of ransomware and cryp-
tojacking since 2017 [3, 4, 5].
In a fileless attack, there is no file to inspect, but commands and network
activity are detectable. Fileless cryptojacking resides in memory (RAM)
without touching the disk on a computer. In Windows systems, PowerShell
plays a significant role in these advanced volatile attacks with its exploitable
features [6, 5].
PowerShell-based fileless cryptojacking attacks can be initiated with phish-
ing emails, zero-day vulnerability exploitations, or hidden scripts on mali-
cious websites [7]. Malicious scripts are often stored and downloaded from
text storage web services, such as Pastebin [8], as observed in fileless ran-
somware attacks [9] [10].
PowerShell can access critical Windows system functions with remote
content retrieval, in-memory command executions, and access to local reg-
istry keys and scheduled tasks [4]. Malware authors exploit it by interacting
with the operating system and almost all Microsoft software, from the Office
suite to the SQL Server database engine. [11, 12].
Its adaptability helps threat actors reduce the need to customize payloads
or download overtly malicious tools to infected systems. They can filelessly
load and execute commands using its invocation feature without touching
the disk or creating new system processes [13].
With this fileless nature, even if attacks are detected, and the original
malicious scripts are identified and removed, the processes may remain op-
erational at the victim endpoints [5].
In some cases, fileless malware may exhibit indirect file activity while
establishing a mining service or persistence mechanism. However, the initial
phases of the attacks are still fileless [14].
As seen in Fig. 1 and Fig. 2, threat actors deliver malicious Excel doc-
uments that contain malicious office macros to deploy a DLL downloader
via the register server (regsvr32). The macros trigger malicious encoded
PowerShell scripts to conduct process injection and add new registry entry.
Finally, the CobaltStrike payload, an exploitable penetration testing script,
is injected into a legitimate process to establish C2. The encoded payload is

2
 Figure 1: Sample encoded PowerShell-based fileless cryptojacking script.

Figure 2: Sample PowerShell-based fileless cryptojacking attacks workflow.

decoded and run on memory to deploy cryptomining [15, 16, 17].

In fileless attacks such as those involving Cobalt Strike, scripts have been
observed employing obfuscation techniques such as Base64-encoded com-
mands as seen in Figure 2 with GunZip compression and encrypted XOR
keys [18]. These methods hide specific Windows API calls made by the shell-
code and process injection into legitimate processes, and they also mask the
establishment of Command and Control (C2) connections and secure session
data with encryption [19].
With the emergence of new Remote Code Execution (RCE) vulnerabil-
ities, such as Log4Shell (CVE-2021-44228) and SolarWinds Orion (CVE-
2020-10177), which affect numerous software products across various indus-
tries, widespread fileless cryptojacking and ransomware attacks have become
prevalent [20]. Additionally, threat actors use various open-source frame-
works, including PowerShell Empire and PowerSploit, which are designed to
support PowerShell scripting in post-exploitation cyber-offensive activities
[16, 4].
PowerShell-based fileless cryptojacking attacks can involve the creation
of encrypted or decoded registries containing decryption routines, using re-
flective Portable Executable (PE) loaders for executing malicious code, and

3
deploying process injection. These are key features of fileless attacks that
provide sophisticated evasion techniques and persistence mechanisms [21].
In particular, a reflective PE loader is commonly used for process injection.
This technique allows the malware to load and execute its payload directly
into memory, bypassing traditional file-based detection mechanisms [18].
Since fileless threats can provide attackers with command and control
abilities using backdoors [9], a fileless cryptojacking attack can be converted
into a data exfiltration activity.
In the literature, there are various kinds of research on specifically in-
browser and in-host cryptojackings with the detection of their attack pat-
terns, especially in network behavior [22, 23, 24, 25, 1, 2, 26, 27, 28]. Also,
there is various research to identify malicious PowerShell scripts, in general,
using machine learning techniques [29, 30, 31, 32, 4, 33, 34, 35, 36, 37, 38].
However, there is a gap in that no research specifically addressed PowerShell-
based fileless cryptojacking.
Hence, there is a crucial need for research in this domain. To our best
knowledge, no research has been conducted on detecting PowerShell-based
fileless cryptojacking using machine learning. This gap can be filled out with
a unique dedicated dataset derived from those attacks.
We attempted to fill this research gap by collecting a unique dataset and
applying machine learning models with an experimental study on detect-
ing PowerShell-based fileless cryptojacking scripts. The results showed that
Abstract Syntax Tree (AST)-based fine-tuned CodeBERT achieved a high
recall rate, proving the importance of using AST integration and fine-tuned
pre-trained models for programming language.

2. Method
The research body on fileless malware detection have used machine learn-
ing models to identify malicious PowerShell scripts [29, 30, 31, 32, 4, 33, 34,
35, 36, 37, 38]. Researchers commonly used deep learning techniques with
natural language processing approaches [35, 39]. However, these solutions try
to detect general malicious scripts. There is a need in the literature that ma-
chine learning models can also be used to improve detections on specifically
PowerShell-based fileless cryptojacking attacks.
In this paper, we use a new mixed method for code representations of Pow-
erShell scripts with their Abstract Syntax Trees (ASTs) using the selected
machine learning models, Long Short-Term Memory (LSTM), Bidirectional

4
LSTM (BiLSTM), and CodeBERT. The goal is to prepare this mixed ap-
proach for model training to detect cryptojacking scripts with other general
PowerShell scripts. The assumption is that the mixed method provides ef-
fective detection mechanisms for detecting cryptojacking scripts as well as
other general malicious PowerShell scripts.
Furthermore, the assumption is that developing light and fast-based de-
tection models instead of relying on manually designed features and specific
feature extraction can help the detections rather than complex approaches
such as feature extractions and de-obfuscation processes.
First, we conduct a background literature review on the existing and latest
solutions to prepare an experimental base for machine learning models. This
phase examines Abstract Syntax Tree (AST) representations of PowerShell
scripts that can be useful to prepare data inputs for training in the selected
machine learning models. We also reviewed existing models for this domain
to prepare and select the appropriate approach for machine learning models.
Second, we collected a dataset that contains 500 collected malicious Pow-
erShell based fileless cryptojacking scripts. Those scripts include Purple Fox,
Lemon Duck, and Tor2Mine malware families. We merged our dataset with a
secondary dataset [36] that contains 1,770 malicious scripts and 4,819 benign
scripts but does not specifically target cryptojacking. Because cryptojacking
also embeds general malicious PowerShell scripts to carry out attacks. We
used those datasets to train and evaluate machine learning models.
In addition, we followed the cleaning and pre-processing steps as proper
inputs for machine learning algorithms. The selected machine learning mod-
els were trained and evaluated.
The results were analyzed to determine the effectiveness of each model
in detecting fileless cryptojacking scripts. We evaluate both AST-based and
non-AST-based approaches to see the effectiveness of this mixed approach.
Instead of heavily modifying models, focusing on feature extraction, or under-
going the complex process of obfuscation-deobfuscation examinations with an
AST-based approach provides flexibility. This flexible approach is powered
by an AST-pipeline-based examination with minimally modified versions of
common machine learning models.
Since it is hard to find malicious samples and easy to find benign samples,
the number of benign samples is higher. However, the ratio is not heavily
skewed. This helps ensure that the machine learning models receive a fair
representation of both categories, reducing bias.
The comparison of file sizes shows an overlapping distribution, and mali-

5
cious scripts have a slightly lower median file size compared to benign ones.
Some malicious scripts are significantly larger due to obfuscation or all-in-one
script features. Both benign and malicious scripts have similar lengths with a
concentration below 500 lines. Malicious scripts have a slightly wider spread
due to longer scripts. Malicious script’s entropy is higher due to more obfus-
cation, encoding, "for loops," and "while loops" in the longer scripts. This
also shows a pattern that benign scripts commonly contain more readable
and structured content than malicious scripts.

3. Background Literature
In the following, we conduct a review of the background literature on the
existing and latest solutions to prepare an experimental base for machine
learning models. This phase examines Abstract Syntax Tree (AST) repre-
sentations of PowerShell scripts that can be useful to prepare data inputs for
training in the selected machine learning models. We also reviewed existing
models for this domain to prepare and select the appropriate approach for
machine learning models.

3.1. Abstract Syntax Tree (AST)

Abstract Syntax Tree (AST), a tree-like version of the code to show the
abstract syntactic structure of the source code [40], can be used for malicious
script detection. It is used in both programming languages and software
development tools [41] to clearly explain code analysis [42].
Specifically, AST breaks down the code into its components and organizes
them hierarchically as a tree-style version. Each node in the tree represents
an element of the language with the expressions, statements, or operators
[35]. By analyzing the AST, we can understand the main logic of the code
without looking into the specific syntax or formatting details. This allows for
a more precise evaluation of the code’s potential impacts and risks [43, 42].
AST analysis is useful for identifying patterns, detecting potential vul-
nerabilities, and ensuring compliance with coding standards [35]. It provides
information on the functionality of the code and potential risks as effective
code review and security assessment processes [43, 42].
As shown in Fig. 3, it breaks down a ping command into hierarchical
components. This feature can help machine learning models to learn the
patterns with the hierarchical structure and sequential data.

6
 Figure 3: Sample AST of a Ping command: “ping -c 4 -t 64 [Link]".

3.2. Abstract Syntax Tree (AST) Types

In order to show the types of AST, the sample PowerShell command is
showed to break down. This tree structure shows the root and child nodes
based on the AST types.
The sample command below is a Purple Fox cryptojacking malicious com-
mand for a C2 connection. The PowerShell script downloads a backdoor
written in .NET implanted during the intrusion. The backdoor is FoxSocket,
which is a Purple Fox-related malware that leverages WebSockets to commu-
nicate with its command-and-control (C2) servers. The command downloads
a malicious payload from the specified URLs with an MSI command as an ar-
gument to install the MSI package as an admin without any user interaction
[44].

[Link] -nop -exec bypass -c

IEX (New-Object [Link]).DownloadString(
’http[:]//[Link][:]13405/[Link]’);
MsiMake http[:]//[Link][:]13405/[Link]

The AST types of the sample command above [45]:

• Expression

7
 – CommandExpressionAst: Represents the expression.
– CommandInvocationAst: Invokes the parameters.
– ExpressionAst: Represents the expression invoked.
– TypeNameAst: [Link]
– MethodInvocationAst: Method invocation.
– MethodNameAst: DownloadString
– StringLiteralAst: ’<URL>:13405/[Link]’
– CmdletAst: MsiMake

• Statement

– PipelineAst: Pipeline of commands executed.

– CommandInvocationAst: Execute Invoke-Expression.
– ExpressionAst: Represents IEX.
– MethodInvocationAst: Method call DownloadString.
– ArgumentAst: URL string literal.
– CmdletAst: MsiMake command with its argument.

The Abstract Syntax Tree (AST) tree format:

__________________________________________
PowerShell Cmdlet
Argument: -nop
Argument: -exec bypass
Argument: -c
| IEX (New-Object [Link]).
DownloadString(’PAYLOAD’)
| Invoke-Expression (IEX)
| | MethodCall
| | | New-Object ([Link])
| | | | TypeName: [Link]
| | | Method: DownloadString
| | | | Argument
| | | | | StringLiteral: PAYLOAD
| | Cmdlet or Function Call: MsiMake

8
| | | Argument
| | | | StringLiteral: PAYLOAD
__________________________________________

In order to create the pipeline of ASTs, each PowerShell script is converted

into ASTs using built-in parser class:
[[Link]] of PowerShell [46].
In AST-based machine learning models, the pipeline creation process in-
cludes AST report generation for each malicious and benign script and to-
kenizing them [35, 36, 47, 48, 12, 49]. Also, AST (Abstract Syntax Tree)
pipelines are being examined in a safe environment to handle malicious Pow-
erShell scripts safely.

3.3. Long Short-term Memory (LSTM)

The long short-term memory (LSTM) network is a type of recurrent neu-
ral network (RNN), a type of artificial neural network that processes sequen-
tial data, like text [50]. LSTM is used to predict temporal-dependent data
such as time series [51, 52]. It is outperforms the RNN because it solves the
problem of vanishing gradients [53, 54].
The LSTM networks were proposed by Hochreiter et al. [54] in 1997
and became popular with the successful results on the models of natural
language processing (NLP) with better sentiment analysis and classification.
LSTM approach is capable of capturing high-level abstractions and reducing
dimensionality [55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65].
An LSTM unit consists of a cell, an input gate, an output gate, and a
forget gate [66]. The cell retains information over varying time intervals,
while the three gates control the flow of information. It is a designed model
for classifying sequential data with long-term dependencies. It improved
predictions in text classification [67], making it useful for classifying malicious
and benign scripts.
Fig. 4 shows the LSTM architecture components where each memory cell
state (c) in an LSTM network serves a specific function [52] to capture the
long-term and short-term dependencies in the data. Input gate (i) decides
new information from the input x for the cell state memory (c). Output gate
(o) specifies the output from the memory cell state. Forget gate (f) decides
what to discard from the memory cell state.
Input gates determine which new information to store in the current state,
and output gates assign a value from 0 to 1 to decide which information

9
 Figure 4: The LSTM architecture diagram [68].

from the current state to output. This selective output of relevant informa-
tion helps LSTM networks maintain important long-term dependencies for
making predictions in both current and future time steps [51, 52]. Forget
gates determine which information from the previous state is discarded by
assigning a value between 0 and 1, with 1 indicating that the information
will be retained and zero indicating that it will be discarded [51, 52]. The
output h can be calculated by:

i(t) = σ(Wxi x(t) + Uhi h(t−1) + bi ) (1)

g (t) = tanh(Wxg x(t) + Uhg h(t−1) + bg ) (2)
f (t) = σ(Wxf x(t) + Uhf h(t−1) + bf ) (3)
o (t)
= σ(Wxo x (t)
+ Uho h (t−1)
+ bo ) (4)
c (t)
=f (t)
⊙c (t−1)
+i (t)
⊙g (t)
(5)
h (t) (t)
= tanh(c ) ⊙ q (t)
(6)

where at time t the input, forget, and output gates are i(t) , f (t) , and o(t) ,
respectively. The g (t) , is the input-update value. The memory cell state at
time t is c(t) , and the output of the LSTM unit at time t is h(t) .
With the ability to handle long-term dependencies effectively, LSTMs are
used for developing prediction and classification algorithms that can be used
for better classifying malicious and benign PowerShell script detections.
In the LSTM model, the number of memory units (neurons) in the LSTM

10
layer is defined as a hyperparameter. In order to achieve this goal, LSTM
units work like tiny detectives by looking for different patterns in the script.
Each neuron (detective) can pay attention to different parts of the script,
and they work together as a team to solve the problem. For example, one
can focus on sequences like IEX followed by a suspicious URL. Another one
can look at frequent downloads associated with DownloadString function.
The other one can notice script has Base64 encoding patterns. To prevent
overfitting and to make the model generalized well, the dropout phase acts
as a supervisor to randomly drop some neurons during training. This process
makes the model to focus on a broader range of patterns.

3.4. Bidirectional LSTM (BiLSTM)

The BiLSTM, an improved version of LSTM, processes data in both for-
ward and backward directions. BiLSTM is an effective tool for tasks when
the context from both directions is important, such as malicious script de-
tection [69, 70]. The model can capture both past and future dependencies.
This feature is useful in tasks like text classification [71, 72].
Improving the behavior of two sequence paths with BiLSTM, processing
each row of PowerShell scripts using NLP, and ordering tokenized IDs can
be used to calculate model layer biases and weights in a much more accurate
way.

3.5. Transformers
The transformer model is used to understand and generate human lan-
guage [73, 74] that is useful in applications like chatbots, translation, and
text generation.
The key advantage of Transformers is self-attention [73]. That feature
helps the model to focus on different words in a sentence at the same time.
The model can understand the context rather than their sequential order.
Semantic learning level is higher than other previous models. Older models
read text one word at a time. However, Transformers process entire sentences
at once. This makes the learning faster and more efficient [75, 74, 73, 76].
Transformer-based models include BERT, CodeBERT, GPT, and T5.
Transformers is a neural network architecture for sequence modeling tasks
to get better predictions than RNNs and LSTMs [74]. It was proposed by
Vaswani et al. in 2017 [73] as a trainable attention mechanism for capturing
complex relationships between elements of an input sequence.

11
 It is primarily composed of an encoder-decoder structure, providing a
framework for handling sequences without recurrent computations. They
use positional encodings to incorporate the order of tokens into the model
since they process all tokens in parallel rather than sequentially [74, 73].
Transformers have the Softmax function that plays an important role in
normalizing the attention scores [77, 73].
The attention mechanism focuses on different parts of the input sequence
for predictions. It calculates a weighted sum of the input tokens, where the
weights are determined by the relevance of each part of the input to the
current token [77].

3.6. CodeBERT
Our research focuses on the classification problem of malicious PowerShell
scripts. In this sentimental analysis task, sequence-to-representation models
are useful [78]. CodeBERT can be used [79] as an encoder-only base trans-
former model to analyze scripts sentimentally on specific malicious script
detection.
CodeBERT was developed based on RoBERTa [80] and BERT [81] mod-
els. It uses Transformer-based neural architecture and relies on a bimodal
pre-trained model. It was also designed for both natural language (NL) and
programming language (PL). It can acquire the semantic relationships be-
tween NL and PL. This can provide general-purpose representations that
make the tasks easier, as well as natural language code search and code
documentation generation. [79].
CodeBERT is a proven model with the effectiveness of semantic learn-
ing of code. It is a pre-trained model on both Natural Language (NL) and
Programming Language (PL) that is designed for code-related tasks. Code-
BERT captures semantic meaning better than LSTMs. It is also good at
avoiding overfitting faster due to its pre-training. It is more suitable for
script classification tasks with fine-tunable on specific tasks. Additional fine-
tuning is necessary for training the model because it makes the model better
for specific tasks, such as the classification of benign and malicious scripts.
The results demonstrate that CodeBERT achieves state-of-the-art per-
formance in learning PL and NL tasks [79]. The model is created using the
multi-layer Transformer architecture [73, 79]. CodeBERT is considered ap-
propriate for fundamental classification tasks that rely on syntax analysis
and code execution reports.

12
4. Related Works
Table 1 shows a comparison of the experimental results on malicious
script detections. As seen in the table, malicious script datasets are limited.
Malicious script datasets range from as few as 100 samples [37] to over 6,600
samples [30]. Model training with larger datasets is preferable for accurate
performance evaluation. However, smaller datasets can limit generalization
and suffer from overfitting.
Since high recall is crucial to detect as many malicious scripts as possible,
the focus on recall rate is preferable rather than accuracy. Because, missing
a threat is riskier than false alarms in cyber security.
Feature extraction usage is common. It is used to improve detections;
however, as seen in the papers from [35] and [38], the models still achieve
high recall scores.
The highest recall rate (99.0%) is shown by [32] using XGBoost with fea-
ture extraction. However, they used a relatively small dataset (480 samples),
and they also reported this in the limitations.

5. Preparation of Experimental Analysis

5.1. Pipeline of ASTs
AST reports include command keywords and parameters with their corre-
sponding AST types. However, the current format is not suitable for effective
data preprocessing and NLP tasks because it can reduce model training accu-
racy. Therefore, creating a pipeline for AST reporting is useful so that each
report is clear and ready for preprocessing steps, including tokenization. In
the pipeline, each command and parameter are transformed into pairs with
related AST types, and the command is rewritten per row to file in JSON
format.
First of all, data was labeled as malicious scripts with an integer "1 "
and benign scripts with an integer "0 ". After that, the pipeline processed
the scripts for Abstract Syntax Trees (ASTs). The first phase converted
PowerShell scripts into ASTs using PowerShell’s built-in function:
[[Link]].
Tokenizing is a process in natural language processing (NLP) and ma-
chine learning to break down text data into word or phrase strings. Those
manageable pieces are called tokens. Those meaningful elements help us to
analyze, process, and feed into machine learning models [82].

13
 Table 1: Comparison of the ML results on malicious PowerShell script detections.
Research Malicious Feature Method Recall %
Data Extr
[35] 4,079 No AST-based 85.0
random
forest
[4] 6,290 Yes NLP-based 89.0
CNN
[32] 480 Yes XGBoost 99.0
[36] 2,000 Yes AST-based 97.6
random
forest
[30] 6,609 Yes SdAs- 98.0
based
XGBoost
[37] 100 Yes Random 87.5
forest clas-
sifier
[38] 1,128 No CodeGEN 85.2
+
CodeGPT

Tokenization can be done with two different methods: character-based

or word-based tokenization. We used word-based tokenization in LSTM and
BiLSTM models after the process of AST representations aligning with the
AST approach. The model received the words as tokens. Since this approach
suffers due to out-of-vocabulary (OOV) issues, limiting OOVs is useful.
On the other hand, Transformers has a special tokenization method that
they can use subword-based tokenization such as Byte Pair Encoding (BPE),
WordPiece, or SentencePiece [74].
In this study, we focus on the most frequent words in the dataset and
set a limit of 6,000 unique tokens out of 250,000 tokens. Tokens exceeding
this limit are categorized as “Out of Vocabulary” (OOV) and assigned a
default ID value of 1. “OOV” (Out-Of-Vocabulary) token is used for unknown
words for unseen data. Both benign and malicious scripts are included in the
tokenizer’s vocabulary to provide a comprehensive base for model training.

14
 To improve the effectiveness of the tokenization process, we exluded sys-
tem administration keywords. This filtering helps to avoid the addition of
very common system administrator words into the vocabulary.

5.2. Phases of Machine Learning Models

In order to conduct machine learning analysis, we followed the main
phases. First, we import data by reading and loading the AST data. After
that, we tokenize the data with padding sequences and limiting vocabulary.
Dataset splitting process is defined for each model for the appropriate anal-
ysis. After fine-tuning with adjustment dropouts and hypermeter tuning, we
trained the models.
Since high recall is crucial to detecting as many malicious scripts as pos-
sible, we focused on recall rate rather than accuracy because missing a threat
is riskier than false alarms in cybersecurity.

5.3. Data Splitting and Validation Method

We used traditional data splitting for the LSTM and BiLSTM models
with 70% training, 15% validation, and 15% test data as shown in table 2.

Table 2: LSTM and BiLSTM – Traditional Data Splitting.

Set Total Samples Benign [0] Malicious [1] Token
Training 70% 2,800 1,400 1,400 400
Validation 15% 600 300 300 400
Test 15% 600 300 300 400

Also, we used K-fold cross-validation in the CodeBERT model with 80%

training and 20% validation sets. In K-fold cross-validation, the data is
divided into K parts, and each fold validates on a different 20% of the data
while training on the remaining 80%. Therefore, there is no need to specify
a separate testing set. Because each fold acts as both training and validation
sets as seen in Table 3.

5.4. Model Definition - LSTM and BiLSTM models

The LSTM (Long Short-Term Memory) and BiLSTM (Bidirectional LSTM)
models are proposed with a custom-built model using the embedding layer,
LSTM, dropout, and dense layers.

15
 Table 3: CodeBERT K-5 Cross-Validation
Set Total Samples Benign [0] Malicious [1] Token
Training 80% 3,200 1,600 1,600 400
Validation 20% 800 870 300 400

The embedding layer dimension is 128, and the tokens are represented
with 128 vector dimensions. For example, if a contains the word “Download-
String”, it is first converted into a numeric token such as “295” as if it is the
245th word (token) in the vocabulary. After that, it is transformed into a 128-
dimensional vector with learned weights such as [0.12,-0.56,0.89,...,0.34](128
values); note that the vocabulary size is 6,000 tokens. The number of train-
able word embeddings is 768,000 (6,000 × 128). The sequence length is 400 to
ensure all input sequences have the same length, avoiding exceeding padding.
In the hidden LSTM layer, the number of units is 64 with a balanced
learning capacity and expected computational efficiency. Since the problem
is classification of malicious an benign scripts, the return sequences setting
is defined as false. This gives outputs only the last hidden state focusing on
sequence-level understanding.
The dropout layer is set up with 50% dropout rate to reduce overfitting
by randomly disabling 50% of neurons during training. This step helps for
generalizated model creation. The activation is ReLU in this layer that it
provides LSTM’s output into a higher-dimensional space before final classi-
fication. Also, we used a second drpout layer to prevent overfitting before
final output.
This approach is also applied to the BiLSTM model, that it uses two-
direction learning compared with LSTM learning. That makes BiLSTM a
heavier model training process than LSTM. The adoption of bidirectional
LSTM models offers distinct advantages over conventional LSTM models
[83, 84, 54]. The BiLSTM architecture processes the sequence in two direc-
tions, allowing the model to capture both past and future dependencies [70].
This improves performance on tasks where context matters in both aspects.
This feature is useful in tasks such as sequence classification, named entity
recognition, and machine translation [71].

16
5.5. Model Definition - CodeBERT
As explained in the background section, CodeBERT is a sequence-to-
representation transformer model designed for a variety of text-based tasks,
including the classification of labeled texts for model training [79].
CodeBERT model is proposed as a fine-tuned pre-trained model. Each
parameter in the CodeBERT model is carefully chosen to build a better
model for high recall rate.
First, the model uses RoBERTa-based tokenization based on Hugging
Face model build. It breaks words into sub-word units for better represen-
tation. Also, it uses "Out-of-Vocabulary (OOV)" words by splitting rare or
unknown words into smaller components.
The learning rate is "0.00002" because to prevent sudden drastic changes
during training. As a real world example, small steps are safer while learning
walking. Therefore, this low learning rate helps the model to avoid dramatic
forgetting.
The batch size is 8 to process a small number of scripts at once. In
each iteration, the checking process on the small number of scripts provides
memory efficiency with limited resources.
The weight decay value is 0.01 to avoid too much memorizing everyting
and to much attention on a pattern that can cause overfitting. This process
adds a penalty for large weights ensuring generalized model creation.
The model uses AdamW (Adaptive Moment Estimation with Weight De-
cay) optimizer to clean up learning to focus on the important patterns in
classification. This process also helps us prevent overfitting.
As a loss function, the model uses "Cross-Entropy Loss" that is a typical
function for classification problems. This helps the model to learn proba-
bilistic outputs. As a real world example, this function acts as a supervisor
and uses a scoreboard to check the validations.
We use K-Fold Cross-Validation (K=5) for model training accuracy val-
idation to monitor overfitting. The model is tested across multiple splits of
data. This process reduces dependency on a single train-test split. It can
detect overfitting and assess generalization.
The model has 12 transformer layers and 12 attention heads for better
self-attention mechanisms. Each layer processes hierarchical representations,
checking complex patterns in the malicious and benign scripts. The model
can focus on different parts simultaneously with the attention heads.
In this model, the hidden dimensionality size is 768. Each token is rep-
resented as a 768-dimensional vector in transformer layers.

17
 Figure 5: BiLSTM model training and validation loss and accuracy results.

Unlike the use of tokenizers in LSTM-based models, the pre-trained Code-

BERT model is applied to modified AST reports using its pre-trained trans-
former tokenizer. Padding for each array is kept to the maximum length of
a specific AST command block, and truncation is applied as needed [79].
As noted in the literature review on CodeBERT [79], the main training
application, using five main code syntaxes and functionalities, improves the
efficiency of the built-in dataset by accommodating different syntax varia-
tions in scripts.
Therefore, fine-tuning the pre-trained CodeBERT model on PowerShell-
based AST reports mitigates the complexity arising from different code rep-
resentations within the dataset while also improving the accuracy of script
classification.

6. Experimental Results

Table 4: Performance Metrics for Related Works and Our Examination.

Model Acc. % Precision % Recall % F1-Score
Non-AST LSTM 90.1 94.5 93.5 91.8
AST-based LSTM 93.3 98.5 95.3 93.8
AST-based BiLSTM 95.3 92.6 91.8 92.2
Non-AST CodeBERT 96.2 96.1 93.6 96.3
AST-based CodeBERT 96.2 96.1 96.6 96.3

The models were evaluated using the fundamental metrics of accuracy,

18
precision, recall, and F1 score. However, we focused on recall scores. Recall
assesses the model’s ability to capture all relevant items within the dataset
to measure how successful the model is against false negatives that can be
critical errors in cybersecurity.
The models were evaluated using fundamental metrics: accuracy, pre-
cision, recall, and F1 score. However, we focused on the recall score tto
measure how well the model performs against false negatives. Detecting ma-
licious scripts as benign scripts is a critical error in cybersecurity domain
that it might allow threat actors access the networks.
The BiLSTM model was trained for over 100 epochs to monitor the over-
fitting. The overfitting signals are observed at the end of the 15th epoch, and
early stopping was applied at the 16th epoch. The model was saved in the
best result with the feature of early stopping. The training and validation
loss and accuracy scores is shown in Fig. 5.
Unlike BiLSTM, in the training of the conventional LSTM model, early
stopping reached the 71th epoch of the training process for the expected range
of validation loss and validation accuracy. LSTM’s best result was also saved
with the feature of early stopping.
The LSTM, BiLSM, and CodeBERT models were trained with AST-
based tokenization and Non-AST tokenization to check the effectiveness of
the AST-based approach as well.
Table 4 shows the performance metrics of three different models with the
AST-based approach options.

7. Key Findings
The results indicates that AST-based fine-tuned CodeBERT achieved a
high recall rate of 96.6% proving the usage of AST can improve the recall
score of CodeBERT model.
Also, the results showed that AST-based fine-tuned CodeBERT can be
used for an effective model with the proposed fine-tuning method without
feature extraction. This can help us reduce computational overhead.
The proposed model provides a better solution compared the other solu-
tions in the literature even though there are other higher recall rates. Because
AST-based fine-tuned CodeBERT achieved a high recall rate of 96.6% with-
out feature extraction and using a balanced dataset with the pre-trained
model.

19
 Although our model does not achieve top results compared to other mod-
els, it improves performance in terms of efficiency. Also, while setting up fine-
tuned model, we used optimized hyperparameters especially for the model
layer dimensions to avoid higher computational need.
Furthermore, our approach does not include obfuscation and de-obfuscation
processes that can cause more processing steps by bringing more computa-
tional overhead. The current models received the tokens as is in terms of
complexity and entropy. This approach also improved efficiency while hav-
ing a high recall rate.

8. Conclusion
This study conducted an experimental study on detecting PowerShell-
based fileless cryptojacking scripts. The experimental results showed that
AST-based fine-tuned CodeBERT achieved a high recall rate of 96.6%. This
result proved the importance of using AST integration. The model achieves
this high recall score without feature extraction. This approach helped us to
reduce computational overhead while maintaining effectiveness.
Although some other models in the literature provided higher recall rates,
our approach brought a more efficient solution. This was achieved with a
balanced dataset and an AST-based fine-tuned pre-trained model. The pre-
trained CodeBERT model was optimized with appropriate hyperparameters,
minimizing computational needs. Additionally, our method avoided obfusca-
tion and de-obfuscation processes. This approach reduced complexity while
maintaining a high recall rate.
As a limitation of this study, dataset creation is extremely challenging
due to the difficulty of accessing malicious scripts. This can be solved by
joint collaboration between academic researchers and security engineers in
cybersecurity services.

9. Acknowledgements
We would like to thank Talha Aydin, a machine learning researcher in
the Department of Computer Science and Information Systems at De Anza
College, California, for providing valuable feedback.

20
10. Submission Declaration
• The work described has not been published previously except in the
form of a preprint, an abstract, a published lecture, academic thesis or
registered report.

• The article’s publication is approved by all authors and tacitly or ex-

plicitly by the responsible authorities where the work was carried out.

• If accepted, the article will not be published elsewhere in the same form,
in English or in any other language, including electronically, without
the written consent of the copyright-holder.

11. Authorship
All authors have made substantial contributions including conception and
design of the study, drafting the article and perform fully revision of the
content.

12. Declaration of Interests

The authors declare that they have no known competing financial inter-
ests or personal relationships that could have appeared to influence the work
reported in this paper.

13. Funding
This research did not receive any specific grant from funding agencies in
the public, commercial, or not-for-profit sectors.

14. Declaration of Generative AI Use

During the preparation of this work the author(s) used Grammarly in
order to assist in spell-check and grammar-check. The author reviewed, val-
idated, and edited all content to ensure accuracy, integrity, and alignment
with scholarly standards and take(s) full responsibility for the content of the
published article.

21
15. Data Statement
Due to the sensitive nature of the collected data that contains malicious
PowerShell scripts, the data would be shared only based on request and
providing signing a responsibility usage form by the requester that includes
the purpose of use and plan. After reviewing and proceed thought the office
of research the access may or may not be provided depending on the office
of research decision.

References
[1] A. Z. Chahoki, H. R. Shahriari, M. Roveri, Cryptojackingtrap: An
evasion resilient nature-inspired algorithm to detect cryptojacking mal-
ware, IEEE Transactions on Information Forensics and Security (2024)
1–1doi:10.1109/TIFS.2024.3353072.

[2] A. Zareh, H. R. Shahriari, Botcointrap: Detection of bitcoin miner bot-

net using host based approach, in: 2018 15th International ISC (Iranian
Society of Cryptology) Conference on Information Security and Cryp-
tology (ISCISC), 2018, pp. 1–6. doi:10.1109/ISCISC.2018.8546867.

[3] M. N. Olaimat, M. A. Maarof, B. A. S. Al-rimy, Ransomware anti-

analysis and evasion techniques: A survey and research directions, in:
2021 3rd international cyber resilience conference (CRC), IEEE, 2021,
pp. 1–6.

[4] D. Hendler, S. Kels, A. Rubin, Detecting malicious powershell com-

mands using deep neural networks, in: Proceedings of the 2018 on Asia
conference on computer and communications security, 2018, pp. 187–
197.

[5] S. Varlioglu, N. Elsayed, Z. ElSayed, M. Ozer, The dangerous combo:

Fileless malware and cryptojacking, SoutheastCon 2022 (2022) 125–132.

[6] A. Bulazel, B. Yener, A survey on automated dynamic malware analysis

evasion and counter-evasion: Pc, mobile, and web, in: Proceedings of
the 1st Reversing and Offensive-oriented Trends Symposium, 2017, pp.
1–21.

22
 [7] P. A. Macaraeg, Arvin Roi; Roderno, [Link].b (2019).
URL [Link]
threat-encyclopedia/malware/[Link].b

[8] L. Gundert. [link].

URL [Link]
[Link]

[9] R. Moussaileb, N. Cuppens, J.-L. Lanet, H. L. Bouder, A survey on

windows-based ransomware taxonomy and detection mechanisms, ACM
Computing Surveys (CSUR) 54 (6) (2021) 1–36.

[10] R. Nataraj, V. Singh, M. Wood, Lemon duck powershell malware cryp-

tojacks enterprise networks (2019).
URL [Link]

[11] A. Rousseau, Hijacking. net to defend powershell, arXiv preprint

arXiv:1709.07508 (2017).

[12] X. Xu, S. Liu, P. Yu, Y. Zhao, Research on powershell obfuscation tech-

nology based on abstract syntax tree transformation, in: 2021 Inter-
national Symposium on Computer Technology and Information Science
(ISCTIS), IEEE, 2021, pp. 121–125.

[13] R. Canary, 2023 red canary threat detection report (Jun 2023).
URL [Link]
threat-detection-report/

[14] Microsoft, Fileless threats (2021).

URL [Link]
threat-protection/intelligence/fileless-threats

[15] J. White, Pulling back the curtains on encodedcommand powershell

attacks (Mar. 2017).
URL [Link]
unit42-pulling-back-the-curtains-on-encodedcommand-powershell-attacks

[16] V. FANG, Malicious powershell detection via machine learning (Jul

2018).
URL [Link]
malicious-powershell-detection-via-machine-learning

23
[17] T. H. N. Newsroom, Hackers use ms excel macro to launch multi-stage
malware attack in ukraine (Jun 2024).
URL [Link]
[Link]

[18] S. Varlioglu, N. Elsayed, E. R. Varlioglu, M. Ozer, Z. ElSayed, The

pulse of fileless cryptojacking attacks: Malicious powershell scripts, in:
SoutheastCon 2024, IEEE, 2024, pp. 571–580.

[19] P. Newton, Analysing fileless malware: Cobalt strike beacon (Jul 2020).
URL [Link]

[20] G. S. Gabor Szappanos, S. Gallagher, Horde of miner bots and backdoors

leveraged log4j to attack vmware horizon servers (Mar 2022).
URL [Link]

[21] M. Ang, B. Gelera, M. Villanueva, Fileless cryptocurrency mining mal-

ware uses new technique, additional variant already spotted (Sep 2018).
URL [Link]

[22] G. Hong, Z. Yang, S. Yang, L. Zhang, Y. Nan, Z. Zhang, M. Yang,

Y. Zhang, Z. Qian, H. Duan, How You Get Shot in the Back: A System-
atical Study about Cryptojacking in the Real World, in: ACM SIGSAC
Conference on Computer and Communications Security, ACM, 2018,
p. 13. doi:10.1145/3243734.3243840.
URL [Link]

[23] J. D. Parra Rodriguez, J. Posegga, RAPID: Resource and API-Based

Detection Against In-Browser Miners, in: ACM 34th Annual Computer
Security Applications Conference, ACM, 2018. doi:10.1145/3274694.
3274735.
URL [Link]

[24] R. K. Sachan, R. Agarwal, S. K. Shukla, Dns based in-browser cryp-

tojacking detection, in: 2022 Fourth International Conference on
Blockchain Computing and Applications (BCCA), IEEE, 2022, pp. 259–
266.

[25] M. Saad, A. Khormali, A. Mohaisen, End-to-End Analysis of In-Browser

Cryptojacking, arXiv preprint arXiv:1809.02152. (sep 2018). arXiv:

24
 1809.02152.
URL [Link]

[26] M. H. Khan Abbasi, S. Ullah, T. Ahmad, A. Buriro, A real-time hybrid

approach to combat in-browser cryptojacking malware, Applied Sciences
13 (4) (2023) 2039.

[27] M. A. Razali, S. Mohd Shariff, Cmblock: In-browser detection and

prevention cryptojacking tool using blacklist and behavior-based detec-
tion method, in: International Visual Informatics Conference, Springer,
2019, pp. 404–414.

[28] F. N. Naseem, A. Aris, L. Babun, E. Tekiner, A. S. Uluagac, Minos: A

lightweight real-time cryptojacking detection system., in: NDSS, 2021.

[29] J. Lu, R. Smith, A heuristic approach to signature based detection of

powershell obfuscation.

[30] A. Alahmadi, N. Alkhraan, W. BinSaeedan, Mpsautodetect: a mali-

cious powershell script detection model based on stacked denoising auto-
encoder, Computers & Security 116 (2022) 102658.

[31] J. Song, J. Kim, S. Choi, J. Kim, I. Kim, Evaluations of ai-based ma-

licious powershell detection with feature optimizations, ETRI Journal
43 (3) (2021) 549–560.

[32] M. Mimura, Y. Tajiri, Static detection of malicious powershell based on

word embeddings, Internet of Things 15 (2021) 100404.

[33] A. Rubin, S. Kels, D. Hendler, Amsi-based detection of mali-

cious powershell code using contextual embeddings, arXiv preprint
arXiv:1905.09538 (2019).

[34] H.-H. Hung, J.-L. Chen, Y.-W. Ma, Machine learning approaches to
malicious powershell scripts detection and feature combination analysis,
Journal of Internet Technology 25 (1) (2024) 167–173.

[35] G. Rusak, A. Al-Dujaili, U.-M. O’Reilly, Ast-based deep learning for de-
tecting malicious powershell, in: Proceedings of the 2018 ACM SIGSAC
Conference on Computer and Communications Security, 2018, pp. 2276–
2278.

25
[36] Y. Fang, X. Zhou, C. Huang, Effective method for detecting malicious
powershell scripts based on hybrid features, Neurocomputing 448 (2021)
30–39.

[37] O. Khalid, S. Ullah, T. Ahmad, S. Saeed, D. A. Alabbad, M. Aslam,

A. Buriro, R. Ahmad, An insight into the machine-learning-based fileless
malware detection, Sensors 23 (2) (2023) 612.

[38] P. Liguori, C. Marescalco, R. Natella, V. Orbinato, L. Pianese, The

power of words: Generating powershell attacks from natural language,
arXiv preprint arXiv:2404.12893 (2024).

[39] X. Yang, G. Peng, D. Zhang, Y. Gao, C. Li, Powerdetector: Malicious

powershell script family classification based on multi-modal semantic
fusion and deep learning, China Communications 20 (11) (2023) 202–
224.

[40] R. F. Crew, et al., Astlog: A language for examining abstract syntax

trees., in: DSL, Vol. 97, 1997, p. 18.

[41] J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, X. Liu, A novel

neural source code representation based on abstract syntax tree, in:
2019 IEEE/ACM 41st International Conference on Software Engineer-
ing (ICSE), IEEE, 2019, pp. 783–794.

[42] C. J. Holz, Investigating representations of obfuscated malicious power-

shell, Ph.D. thesis, Massachusetts Institute of Technology (2019).

[43] K. D. Cooper, L. Torczon, Chapter 4 - intermediate repre-

sentations, in: K. D. Cooper, L. Torczon (Eds.), Engineer-
ing a Compiler (Third Edition), third edition Edition, Mor-
gan Kaufmann, Philadelphia, 2023, pp. 159–207. doi:https:
//[Link]/10.1016/B978-0-12-815412-0.00010-3.
URL [Link]/science/article/pii/
B9780128154120000103

[44] A. Sharshar, J. Yaneza, S. & Magdy, Purplefox adds new backdoor that
uses websockets (2021).

26
[45] M. Learn, Ast types.
URL [Link]
[Link]

[46] C. M. Schubert Kabban, S. R. Graham, W. C. Henry, C. M. Ron-

deau, et al., Malware classification through abstract syntax trees and
l-moments, Computers & Security (ISSN 0167-4048) (2024).

[47] H. Nurmi, et al., Script and macro based malware classification by clus-
tering abstract syntax trees, Master’s thesis (2019).

[48] A. J. Rose, S. R. Graham, C. M. Schubert Kabban, J. J. Krasnov, W. C.

Henry, Scriptblock smuggling: Uncovering stealthy evasion techniques in
powershell and. net environments, Journal of Cybersecurity and Privacy
4 (2) (2024) 153–166.

[49] H. Miao, H. Bao, Z. Tang, W. Li, W. Wang, H. Chen, F. Liu, Y. Sun,

Ast2vec: A robust neural code representation for malicious powershell
detection, in: International Conference on Science of Cyber Security,
Springer, 2023, pp. 207–224.

[50] N. Elsayed, S. Anthony, M. Bayoumi, Deep gated recurrent and convo-

lutional network hybrid model for univariate time series classification,
International Journal of Advanced Computer Science and Applications
10 (5) (2019).

[51] N. Elsayed, Gated convolutional recurrent neural networks for predictive

coding, University of Louisiana at Lafayette, 2019.

[52] S. W. Azumah, N. Elsayed, V. Adewopo, Z. S. Zaghloul, C. Li, A deep

lstm based approach for intrusion detection iot devices network in smart
home, in: 2021 IEEE 7th World Forum on Internet of Things (WF-IoT),
IEEE, 2021, pp. 836–841.

[53] N. Elsayed, A. S. Maida, M. Bayoumi, Reduced-gate convolutional long

short-term memory using predictive coding for spatiotemporal predic-
tion, Computational Intelligence 36 (3) (2020) 910–939.

[54] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural com-

putation 9 (8) (1997) 1735–1780.

27
[55] B. Alshemali, J. Kalita, Improving the reliability of deep neural networks
in nlp: A review, Knowledge-Based Systems 191 (2020) 105210.

[56] M. Gimenez, J. Palanca, V. Botti, Semantic-based padding in convo-

lutional neural networks for improving the performance in natural lan-
guage processing. a case of study in sentiment analysis, Neurocomputing
378 (2020) 315–323.

[57] J. Guo, H. He, T. He, L. Lausen, M. Li, H. Lin, X. Shi, C. Wang, J. Xie,
S. Zha, et al., Gluoncv and gluonnlp: Deep learning in computer vision
and natural language processing, Journal of Machine Learning Research
21 (23) (2020) 1–7.

[58] H.-j. Park, M. Song, K.-S. Shin, Deep learning models and datasets
for aspect term sentiment classification: Implementing holistic recurrent
attention on target-dependent memories, Knowledge-Based Systems 187
(2020) 104825.

[59] L. Abualigah, H. E. Alfar, M. Shehab, A. M. A. Hussein, Sentiment

analysis in healthcare: a brief review, Recent advances in NLP: the case
of Arabic language (2020) 129–141.

[60] A. Balamurali, B. Ananthanarayanan, Develop a neural model to score

bigram of words using bag-of-words model for sentiment analysis, Neural
Networks for Natural Language Processing - (2020) 122–142.

[61] T. Fischer, C. Krauss, Deep learning with long short-term memory net-
works for financial market predictions, European journal of operational
research 270 (2) (2018) 654–669.

[62] N. Elsayed, Z. ElSayed, A. S. Maida, Litelstm architecture for deep

recurrent neural networks, in: 2022 IEEE International Symposium on
Circuits and Systems (ISCAS), IEEE, 2022, pp. 1304–1308.

[63] I. J. Unanue, E. Z. Borzeshi, M. Piccardi, Recurrent neural net-

works with specialized word embeddings for health-domain named-entity
recognition, Journal of biomedical informatics 76 (2017) 102–109.

[64] Y. Luo, Recurrent neural networks for classifying relations in clinical

notes, Journal of biomedical informatics 72 (2017) 85–95.

28
[65] H. Jelodar, Y. Wang, R. Orji, S. Huang, Deep sentiment classification
and topic discovery on novel coronavirus or covid-19 online discussions:
Nlp using lstm recurrent neural network approach, IEEE - Biomedical
and Health Informatics 24 (10) (2020).
[66] F. A. Gers, J. Schmidhuber, F. Cummins, Learning to forget: Continual
prediction with lstm, Neural computation 12 (10) (2000) 2451–2471.
[67] S. Hochreiter, J. Schmidhuber, Lstm can solve hard long time lag prob-
lems, Advances in neural information processing systems 9 (1996).
[68] N. Elsayed, Z. ElSayed, A. S. Maida, Litelstm architecture based
on weights sharing for recurrent neural networks, arXiv preprint
arXiv:2301.04794 (2023).
[69] A. Graves, J. Schmidhuber, Framewise phoneme classification with bidi-
rectional lstm and other neural network architectures, Neural networks
18 (5-6) (2005) 602–610.
[70] A. Graves, A.-r. Mohamed, G. Hinton, Speech recognition with deep
recurrent neural networks, in: 2013 IEEE international conference on
acoustics, speech and signal processing, Ieee, 2013, pp. 6645–6649.
[71] N. Elsayed, Z. S. Zaghloul, S. W. Azumah, C. Li, Intrusion detection
system in smart home network using bidirectional lstm and convolu-
tional neural networks hybrid model, in: 2021 IEEE International Mid-
west Symposium on Circuits and Systems (MWSCAS), 2021, pp. 55–58.
doi:10.1109/MWSCAS47672.2021.9531683.
[72] M. Boden, A guide to recurrent neural networks and backpropagation,
the Dallas project 2 (2) (2002) 1–10.
[73] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural
information processing systems 30 (2017).
[74] C. Wang, M. Li, A. J. Smola, Language models with transformers, arXiv
preprint arXiv:1904.09408 (2019).
[75] N. K. Manaswi, N. K. Manaswi, Rnn and lstm, Deep learning with appli-
cations using python: chatbots and face, object, and speech recognition
with TensorFlow and Keras (2018) 115–126.

29
[76] D. Soydaner, Attention mechanism in neural networks: where it comes
and where it goes, Neural Computing and Applications 34 (16) (2022)
13371–13385.

[77] Z. Niu, G. Zhong, H. Yu, A review on the attention mechanism of deep

learning, Neurocomputing 452 (2021) 48–62.

[78] C. Dos Santos, M. Gatti, Deep convolutional neural networks for senti-
ment analysis of short texts, in: Proceedings of COLING 2014, the 25th
international conference on computational linguistics: technical papers,
2014, pp. 69–78.

[79] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin,

T. Liu, D. Jiang, et al., Codebert: A pre-trained model for programming
and natural languages, arXiv preprint arXiv:2002.08155 (2020).

[80] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis,

L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pre-
training approach, arXiv preprint arXiv:1907.11692 (2019).

[81] J. Devlin, Bert: Pre-training of deep bidirectional transformers for lan-

guage understanding, arXiv preprint arXiv:1810.04805 (2018).

[82] S. Kul, A. Manga, Z. Esenalp, R. Kaplan, A. Sayar, Detecting malicious

and clean powershell scripts among obfuscated commands using deep
learning methods and word embedding, in: The Proceedings of the In-
ternational Conference on Smart City Applications, Springer, 2022, pp.
859–868.

[83] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer,

Neural architectures for named entity recognition, arXiv preprint
arXiv:1603.01360 (2016).

[84] H. T.-T. Do, H. D. Huynh, K. Van Nguyen, N. L.-T. Nguyen, A. G.-T.

Nguyen, Hate speech detection on vietnamese social media text using
the bidirectional-lstm model, arXiv preprint arXiv:1911.03648 (2019).

30