As programming has matured over time, modular application design has increasingly been given due (and desired) consideration. Often, especially for application programmers, this means focusing on simple, modular, and flexible design without giving much consideration to security. For example, if we talk about a procedural language like C, functions call each other under a master deriving function, main. Whether a function is able to do what it promises or not is signaled through a return value, which can provide decision-making information in the application later on, and which, in turn, can change the application flow.

The Modify-function-return-value hack changes the return value of a function in a manner that can be used by a hacker for his advantage. It is not necessary to apply this hack to each and every function in the application/library. A key function, if hacked, may compromise the whole application — this is highly dependent on the application design itself.
In this article, I explain how this hack can be executed and what a developer can do to make such attacks harder for the hacker. I used the word "harder" because there is no 100% guaranteed defense.

An Example Victim Application

Let's look at how an application is vulnerable to this kind of hack. For this example, suppose the hacker is attacking an application, we'll call it victim_client. Let's assume the app is running on x86-based Linux.

The hacker, posing as a user, knows that victim_client, once invoked from a computer (say, computer A), does some processing. It relies on some services from a remote server whose IP address it reads from an environment variable 'REMOTE_HOST.' However, the hacker can run this on his computer only; he can't use this application on another computer (say, computer B). Here is the crux of problem: The hacker wants to run this application on another computer. To solve this problem, he needs some information about the victim application. For the rest of this example, I am assuming the role of a hacker to demonstrate one way of perpetrating the hack. This is just the one way — there are many others.

Information Gathering

Tools like ltrace and strace can provide some initial information to hackers. So, let's execute the victim application, on computer B, under the influence of these commands:

Output of 'ltrace'

[raman@ComputerB]$ export REMOTE_HOST=192.168.109.138
[raman@ComputerB]$ ltrace -s 500 ./victim_client 
(0x49dfa51c, 0x49dfaab0, 0, 0, 0)  = 0x49dfa8e4
__libc_start_main(0x8048863, 1, 0xbfd5bed4, 0x80488e0, 0x8048950 <unfinished ...>
getenv("REMOTE_HOST")              = "192.168.109.138"
gethostname("ComputerB", 255)      = 0
socket(2, 2, 17)                   = 3
htons(0, 2, 17, 0x49e06458, 0x49e00e38) = 0
htonl(0, 2, 17, 0x49e06458, 0x49e00e38) = 0
bind(3, 0xbfd5bdc8, 16, 0x49e06458, 0x49e00e38)= 0
htons(9930, 0xbfd5bdc8, 16, 0x49e06458, 0x49e00e38) = 51750
inet_aton("192.168.109.138", 0xbfd5bddc)        = 1
snprintf("ComputerB", 99, "%s", "ComputerB")    = 9
sendto(3, "ComputerB", 100, 0, 0xbfd5bdd8, 16)  = 100
recvfrom(3, 0xbfd5bd60, 100, 0, 0xbfd5bdd8)     = 30
close(3)                                        = 0
puts("\n Could not initialize... exiting"
Could not initialize... exiting)               = 34
exit(1 <no return ...>
+++ exited (status 1) +++
[raman@ComputerB ARTICLE]$

What is happening here (on lines 7, 14, 15, and 18) is:

• The application is creating a socket to communicate to a remote entity.
• It is sending some data(100 bytes).
• It is receiving some data (30 bytes).
• And then it fails to execute, as expected, because computer B is not supposed to run that application.

Output of 'strace'

[raman@ComputerB]$ strace -s 50 ./victim_client 
execve("./victim_client", ["./victim_client"], [/* 29 vars */]) = 0
brk(0)                                  = 0x8f7b000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7794000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=100980, ...}) = 0
mmap2(NULL, 100980, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb777b000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20h\341I4\0\0\0\4\177\36\0\0\0\0\0004\0 \0\n\0(\0+\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=2000316, ...}) = 0
mmap2(0x49dfd000, 1759836, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x49dfd000
mprotect(0x49fa4000, 4096, PROT_NONE)   = 0
mmap2(0x49fa5000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a7) = 0x49fa5000
mmap2(0x49fa8000, 10844, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x49fa8000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb777a000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb777a6c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x49fa5000, 8192, PROT_READ)   = 0
mprotect(0x49df9000, 4096, PROT_READ)   = 0
munmap(0xb777b000, 100980)              = 0
uname({sys="Linux", node="ComputerB", ...}) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3
bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
sendto(3, "ComputerB\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 100, 0, {sa_family=AF_INET, sin_port=htons(9930), sin_addr=inet_addr("192.168.109.138")}, 16) = 100
recvfrom(3, "STOP\0terB\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 100, 0, {sa_family=AF_INET, sin_port=htons(9930), sin_addr=inet_addr("192.168.109.138")}, [16]) = 30
close(3)                                = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7793000
write(1, "\n", 1)                     = 1
write(1, " Could not initialize... exiting\n", 33 Could not initialize... exiting) = 33
exit_group(1)                           = ?
[raman@ComputerB ]$ 

What is happening here (on lines 26, 27, and 32) is:

• The application is sending some data to IP address 192.168.109.138, using port 9930. The length of the data sent is 100 bytes. However, most of the data bytes are \0. It may be the remote entity is using this all 100 bytes or only the non-null bytes, we're not sure.
• The application receives data — a length of 30 bytes. The received data is STOP\0. Again, most the bytes are \0.
• The application fails with the error code.

Running the application under strace on computer A, I got SUCCESS\0, as the data received in the second parameter to recvfrom. This is interesting information. It's clear that the server is sending different responses to the application when it is running on computer A than when it is running on computer B.