Operating Systems Security
Buffer Overflow Attacks
CSE 565 - Fall 2025
Computer Security
Hongxin Hu (hongxinh@[Link])
Updates
• Project 4 Jailbreaking LLMs
– Deadline:
• Thursday, November 20, 2025
What is an Exploit?
• An exploit is any input (i.e., a piece of software,
an argument string, or sequence of commands)
that takes advantage of a bug, glitch or
vulnerability in order to cause an attack
• An attack is an unintended or unanticipated
behavior that occurs on computer software,
hardware, or something electronic and that
brings an advantage to the attacker
01/13/2026 Buffer Overflow 3
Possible Attack Scenario
• Users enter data into a Web form
• Web form is sent to server
• Server writes data to array called buffer, without
checking length of input data
• Data “overflows” buffer
– Such overflow might enable an attack
– If so, attack could be carried out by anyone with
Internet access
Buffer Overflow
int main(){
int
buffer[10];
buffer[20] =
37;}
• Q: What happens when code is executed?
• A: Depending on what resides in memory at
location “buffer[20]”
– Might overwrite user data or code
– Might overwrite system data or code
– Or program could work just fine
Simple Buffer Overflow
• Consider boolean flag for authentication
• Buffer overflow could overwrite flag allowing
anyone to authenticate
Boolean flag
buffer
F O U R S C … F
T
In some cases, Trudy need not be so lucky
as in this example
Buffer Overflow Attack
• One of the most common OS bugs is a buffer
overflow
– The developer fails to include code that checks
whether an input string fits into its buffer array.
Then, an input to the running process can exceed
the length of the buffer
– The input string overwrites a portion of the
memory of the process and causes the application
to behave improperly and unexpectedly
01/13/2026 Buffer Overflow 7
Buffer Overflow Attack
• Effect of a buffer overflow
– The process can operate on malicious data or
execute malicious code passed in by the attacker
– If the process is executed as root, the malicious
code will be executing with root privileges
01/13/2026 Buffer Overflow 8
Memory Organization
• Process Address Space High Addresses
– Text == code 0xFFFF FFFF
– Data == static variables Stack
– Heap == dynamic data
– Stack == “scratch paper”
• Dynamic local variables Heap
• Parameters to functions
• Return address Data
Text
Low Addresses
0x0000 0000
Vulnerabilities and Attack Method
• Vulnerability scenarios
– The program has root privileges (setuid) and is
launched from a shell
– The program is part of a web application
• Typical attack method
1. Find vulnerability
2. Reverse engineer the program
3. Build the exploit
01/13/2026 Buffer Overflow 10
Buffer Overflow Attack in a Nutshell
• First described in
Aleph One. Smashing The Stack For Fun And Profit. e-zine
[Link] #49, 1996
• The attacker exploits an unchecked buffer to perform a
buffer overflow attack
• The ultimate goal for the attacker is getting a shell that
allows to execute arbitrary commands with high privileges
• Kinds of buffer overflow attacks:
– Stack smashing
– Heap smashing
01/13/2026 Buffer Overflow 11
Buffer Overflow
Top of
Memory
domain.c 0xFFFFFFFF
Main(int argc, char *argv[ ]) Stack
/* get user_input */ Fill
{ Direction
char var1[15];
char command[20];
strcpy(command, “whois "); var1 (15 char)
strcat(command, argv[1]);
strcpy(var1, argv[1]); command
printf(var1); (20 char)
system(command);
} ..
.
• Retrieves domain registration info Bottom of
• e.g., domain [Link] Memory
Buffer Overflow 0x00000000
01/13/2026 12
strcpy() Vulnerability
domain.c Top of
Main(int argc, char *argv[]) Memory
/*get user_input*/ 0xFFFFFFFF Stack
{ Fill
char var1[15]; Direction
char command[20];
strcpy(command, “whois "); argv[1]
var1argv[1]
(15 char)
strcat(command, argv[1]); (15
(20char)
char)
strcpy(var1, argv[1]); Overflow
printf(var1); command
exploit
system(command); (20 char)
}
..
• argv[1] is the user input .
• strcat(d, s) concatenates strings Bottom of
• strcpy(dest, src) does not check buffer Memory
0x00000000
01/13/2026 Buffer Overflow 13
strcpy() vs. strncpy()
• Function strcpy() copies the string in the second
argument into the first argument
– e.g., strcpy(dest, src)
– If source string > destination string, the overflow characters
may occupy the memory space used by other variables
– The null character is appended at the end automatically
• Function strncpy() copies the string by specifying the
number n of characters to copy
– e.g., strncpy(dest, src, n); dest[n] = ‘\0’
– If source string is longer than the destination string, the
overflow characters are discarded automatically
– You have to place the null character manually
01/13/2026 Buffer Overflow
Simplified Stack Example
low
:
void func(int a, int b){ :
char buffer[10];
}
void main(){
func(1, 2); ¬ SP
buffer
}
ret ¬
¬ SP
return
address
a ¬ SP
b ¬ SP
high
Smashing the Stack
low
What happens if ???
:
:
buffer overflows?
Program “returns” to
buffer
¬ SP
wrong location
overflow
ret ¬
¬ret…
SP NOT!
A crash is likely overflow
a ¬ SP
b ¬ SP
high
Smashing the Stack
low
Trudy has a
:
better idea… :
Code injection
(Shellcode) ¬ SP
evil code
Trudy can run
ret
ret ¬ SP
code of her
a ¬ SP
choosing… ¬ SP
b
o …on your machine high
Smashing the Stack
Trudy may not know… :
:
1) Address of evil code NOP
:
2) Location of ret on stack
NOP
Solutions evil code
1) Precede evil code with ret
NOP (No Operation) ret ¬ ret
2) Insert ret many times :
ret
:
:
Stack Smashing Summary
• A buffer overflow must exist in the code
• Not all buffer overflows are exploitable
– Things must align properly
• If exploitable, attacker can inject code (Shellcode )
• Trial and error is likely required
– Fear not, lots of help is available online
– Smashing the Stack for Fun and Profit, Aleph One
• Stack smashing is “attack of the decade”
– Regardless of the current decade
– Also heap overflow, integer overflow, …
Stack Smashing Defenses
• Employ non-executable stack
– “No execute” NX bit (if available)
– Seems like the logical thing to do, but some real code
executes on the stack (Java, for example)
• Address space layout randomization (ASLR)
• Use safe languages (Java, C#)
• Use safer C functions
– For unsafe functions, safer versions exist
– For example, strncpy instead of strcpy
ASLR
• Address Space Layout Randomization
– Randomize place where code loaded in memory
• Makes most buffer overflow attacks probabilistic
• Windows Vista uses 256 random layouts
– So about 1/256 chance buffer overflow works?
• Similar thing in Mac OS X and other OSs
• Attacks against Microsoft’s ASLR do exist
– Possible to “de-randomize”
Stack-based buffer overflow detection
using a random canary
Normal (safe) stack configuration:
Other local Canary Return
Buffer
variables address
Other data
(random)
Buffer overflow attack attempt:
Corrupt
Buffer Overflow data return Attack code x
address
• The canary is placed in the stack prior to the
return address, so that any attempt to over-
write the return address also over-writes the
canary.
01/13/2026 Buffer Overflow 22
Buffer Overflow
• A major security threat yesterday, today, and
tomorrow
• The good news?
– It is possible to reduced overflow attacks
• Safe languages, NX bit, ASLR, education, etc.
• The bad news?
– Buffer overflows will exist for a long time
• Legacy code, bad development practices, etc.