Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼

Web Development

Malicious Work with the modify-function-return-value Hack

As programming has matured over time, modular application design has increasingly been given due (and desired) consideration. Often, especially for application programmers, this means focusing on simple, modular, and flexible design without giving much consideration to security. For example, if we talk about a procedural language like C, functions call each other under a master deriving function, main. Whether a function is able to do what it promises or not is signaled through a return value, which can provide decision-making information in the application later on, and which, in turn, can change the application flow.

The Modify-function-return-value hack changes the return value of a function in a manner that can be used by a hacker for his advantage. It is not necessary to apply this hack to each and every function in the application/library. A key function, if hacked, may compromise the whole application — this is highly dependent on the application design itself.
In this article, I explain how this hack can be executed and what a developer can do to make such attacks harder for the hacker. I used the word "harder" because there is no 100% guaranteed defense.

An Example Victim Application

Let's look at how an application is vulnerable to this kind of hack. For this example, suppose the hacker is attacking an application, we'll call it victim_client. Let's assume the app is running on x86-based Linux.

The hacker, posing as a user, knows that victim_client, once invoked from a computer (say, computer A), does some processing. It relies on some services from a remote server whose IP address it reads from an environment variable 'REMOTE_HOST.' However, the hacker can run this on his computer only; he can't use this application on another computer (say, computer B). Here is the crux of problem: The hacker wants to run this application on another computer. To solve this problem, he needs some information about the victim application. For the rest of this example, I am assuming the role of a hacker to demonstrate one way of perpetrating the hack. This is just the one way — there are many others.

Information Gathering

Tools like ltrace and strace can provide some initial information to hackers. So, let's execute the victim application, on computer B, under the influence of these commands:

Output of 'ltrace'

[[email protected]]$ export REMOTE_HOST=
[[email protected]]$ ltrace -s 500 ./victim_client 
(0x49dfa51c, 0x49dfaab0, 0, 0, 0)  = 0x49dfa8e4
__libc_start_main(0x8048863, 1, 0xbfd5bed4, 0x80488e0, 0x8048950 <unfinished ...>
getenv("REMOTE_HOST")              = ""
gethostname("ComputerB", 255)      = 0
socket(2, 2, 17)                   = 3
htons(0, 2, 17, 0x49e06458, 0x49e00e38) = 0
htonl(0, 2, 17, 0x49e06458, 0x49e00e38) = 0
bind(3, 0xbfd5bdc8, 16, 0x49e06458, 0x49e00e38)= 0
htons(9930, 0xbfd5bdc8, 16, 0x49e06458, 0x49e00e38) = 51750
inet_aton("", 0xbfd5bddc)        = 1
snprintf("ComputerB", 99, "%s", "ComputerB")    = 9
sendto(3, "ComputerB", 100, 0, 0xbfd5bdd8, 16)  = 100
recvfrom(3, 0xbfd5bd60, 100, 0, 0xbfd5bdd8)     = 30
close(3)                                        = 0
puts("\n Could not initialize... exiting"
Could not initialize... exiting)               = 34
exit(1 <no return ...>
+++ exited (status 1) +++
[[email protected] ARTICLE]$

What is happening here (on lines 7, 14, 15, and 18) is:

  • The application is creating a socket to communicate to a remote entity.
  • It is sending some data(100 bytes).
  • It is receiving some data (30 bytes).
  • And then it fails to execute, as expected, because computer B is not supposed to run that application.

Output of 'strace'

[[email protected]]$ strace -s 50 ./victim_client 
execve("./victim_client", ["./victim_client"], [/* 29 vars */]) = 0
brk(0)                                  = 0x8f7b000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7794000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=100980, ...}) = 0
mmap2(NULL, 100980, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb777b000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20h\341I4\0\0\0\4\177\36\0\0\0\0\0004\0 \0\n\0(\0+\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=2000316, ...}) = 0
mmap2(0x49dfd000, 1759836, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x49dfd000
mprotect(0x49fa4000, 4096, PROT_NONE)   = 0
mmap2(0x49fa5000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a7) = 0x49fa5000
mmap2(0x49fa8000, 10844, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x49fa8000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb777a000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb777a6c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x49fa5000, 8192, PROT_READ)   = 0
mprotect(0x49df9000, 4096, PROT_READ)   = 0
munmap(0xb777b000, 100980)              = 0
uname({sys="Linux", node="ComputerB", ...}) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("")}, 16) = 0
sendto(3, "ComputerB\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 100, 0, {sa_family=AF_INET, sin_port=htons(9930), sin_addr=inet_addr("")}, 16) = 100
recvfrom(3, "STOP\0terB\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 100, 0, {sa_family=AF_INET, sin_port=htons(9930), sin_addr=inet_addr("")}, [16]) = 30
close(3)                                = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7793000
write(1, "\n", 1)                     = 1
write(1, " Could not initialize... exiting\n", 33 Could not initialize... exiting) = 33
exit_group(1)                           = ?
[[email protected] ]$ 

What is happening here (on lines 26, 27, and 32) is:

  • The application is sending some data to IP address, using port 9930. The length of the data sent is 100 bytes. However, most of the data bytes are \0. It may be the remote entity is using this all 100 bytes or only the non-null bytes, we're not sure.
  • The application receives data — a length of 30 bytes. The received data is STOP\0. Again, most the bytes are \0.
  • The application fails with the error code.

Running the application under strace on computer A, I got SUCCESS\0, as the data received in the second parameter to recvfrom. This is interesting information. It's clear that the server is sending different responses to the application when it is running on computer A than when it is running on computer B.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.