Channels ▼

Linux Memory Forensics

Linux Memory Forensics

Michael Ford

Forensic analysis is the investigation of an event that involves looking for evidence and interpreting that evidence. In the case of a computer crime in which a system was compromised, the investigator needs to find out who, what, where, when, how, and why.

There are three main areas from which evidence of an intrusion can be gathered. The first and most common is the hard drive. A file system on a hard drive contains the least volatile data. Whether the investigator's strategy involves shutting down the system or just removing the computer's power, the file system will still be there. The investigator's response strategy will dictate what changes are made to the file system. If the file system is shut down or if the investigator issues commands to the system to collect information, the file system may be changed, but in the end, it's still there. There are then many tools, such as The Sleuth Kit or The Coroner's Toolkit (TCT), that can be used to analyze the file system.

The second, and most volatile, of the three areas is network traffic. Once a packet has reached its destination, it's no longer on the wire, and it will only exist briefly in memory on the received system. Regional and national laws dictate the legality of collecting network traffic, but many tools exist to do so. These include intrusion detection systems (IDS), such as Snort, and network monitoring facilities, such as tcpdump or Ethereal.

The third area to investigate is system memory, which I will focus on in this article. The system memory contains everything that is currently happening on the system. If the system is shut down or turned off, this information is lost. We can run many commands before the system is taken down, however, to capture some of this information. For example, we can use date to find the current system time and date. We can use netstat to find a list of the current connections to and from the system and ps to find a list of processes running on the system. But there isn't a command that will show, for example, the changes to files that haven't yet been written to disk. We also cannot be sure that the netstat, date, and ps commands have not been changed by an attacker. An attacker may have modified these and other commands to keep investigators from seeing information about his connections, or he may have replaced them with booby-trapped versions that would erase all of his files, among other things.

Also, running these commands changes their associated access times, which are important when creating a timeline of system activity. Every command that is typed changes the system state. It would be better to save the system memory and analyze it later to find this information. When an attacker breaks into a server and installs a rootkit, many tools can be used to analyze the filesystem of the victim server and find out what the attacker did. Unfortunately, there are far fewer tools and published methods for capturing the memory of an attacked machine and analyzing that image.

One way to save an image of the system memory is to use the crash dump facility. Most Unix operating systems provide a facility for saving the system memory for later analysis. This is useful in cases where a kernel bug causes a crash, and the bug can later be analyzed and fixed. We can use the same facility to analyze the memory after an attack.

I will demonstrate crash analysis to find the answers to three problems. The analysis requires familiarity with the C programming language and some knowledge of Linux kernel structures. To save space, I will omit command output that is irrelevant to the demonstration, and I will replace unneeded structure elements with an ellipsis (...).

The first problem is that, on the victim system, ps does not show all of the processes being run. Originally, ps returned:

[[email protected]]# ps -f
root      1211  1209  0 18:15 pts/0    00:00:00 -bash
root      1266  1211  0 18:22 pts/0    00:00:00 ./malware 41000
root      1267  1211  0 18:22 pts/0    00:00:00 ./invisible 41001
root      1268  1211  0 18:22 pts/0    00:00:00 ./removed 41002
root      1270  1211  0 18:23 pts/0    00:00:00 ps -f
Unfortunately, a trojan ps command was installed to hide any process called "malware". A second problem is that the "adore" Loadable Kernel Module (LKM) was installed. Adore can be used to hide files or processes on the system. Here it was used to make the "invisible" process invisible and to remove the "removed" process from the ps listing forever. So now, ps returns:
[[email protected]]# ps -f
root      1211  1209  0 18:15 pts/0    00:00:00 -bash
root      1281  1211  0 18:24 pts/0    00:00:00 ps -f
The third problem is that the netstat command has been replaced with a trojan version, and the new one doesn't show all the connections or listening ports on the system. Originally, it showed:
[[email protected]]# netstat -an | egrep '(^[AP]|ESTAB|LISTEN)'
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address    Foreign Address   State
tcp        0      0*         LISTEN
tcp        0      0*         LISTEN
tcp        0      0*         LISTEN
tcp        0      0*         LISTEN
tcp        0      0*         LISTEN 
tcp        0      0    ESTABLISHED
and now it only shows:
[[email protected]]# netstat -an | egrep '(^[AP]|ESTAB|LISTEN)'
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address    Foreign Address   State
tcp        0      0*         LISTEN
tcp        0      0*         LISTEN
Creating Crash Dumps with LKCD I will be using the Linux Kernel Crash Dumps (LKCD) package on a Linux 2.4.20 kernel on an X86 system to generate a crash dump. This package consists of three parts: a patch to the kernel, the configuration utility and patches to init scripts, and the lcrash program. To begin, download and install a patch to the kernel to add the source code for the new feature.
# cd /usr/src
# wget
# cd linux-2.4.20
# patch -p1 < ../lkcd-2.4.20.diff
In the kernel configuration file, the CONFIG_MAGIC_SYSRQ and the new CONFIG_DUMP options must be enabled to allow the crash dump to happen. Also, CONFIG_DUMP_COMPRESS_RLE and CONFIG_DUMP_COMPRESS_GZIP are good options to enable so that the dump is compressed to use less disk space. After building the new kernel and rebooting with it, get the LKCD utilities and install them:
# wget
# rpm -i lkcdutils-4.1-1.i386.rpm
You will also want to review the configuration in the "/etc/sysconfig/dump" file to match your system. Changes are also required to the initialization script specified on the "si" line in /etc/inittab. On Red Hat Linux, this file is "/etc/rc.d/rc.sysinit". The file should be changed to run:
/sbin/lkcd config
/sbin/lkcd save
before the swap partitions are activated to keep the dump file from becoming corrupted as the system is booted. To save a dump file, hit a three-key combination (Alt-SysRq-C), and the system will save a dump file to disk, usually to the swap partition specified by the symbolic link to /dev/vmdump. In some cases, the three-key combination above may not work. In that case, you can put the console in sysrq mode by issuing the following command:
[[email protected]]# echo 1 > /proc/sys/kernel/sysrq
Then on the console, hit the "c" key, which will cause a crash and dump. More information on installing and using LKCD can be found on the LKCD Web site listed in the References section of this article. When the system is rebooted, the crash dump will be saved to a file on disk under the directory specified in the configuration file. During a forensic analysis, this is sub-optimal because it would cause major changes to the evidence. It's better to restore the dump from an image of the evidence disk, rather than from the evidence disk itself. One way to do this is to make an image copy of the swap partition containing the dump and then save the dump from this copy. If we write the copy to an unused partition on the forensics workstation, we can save the dump to a file with lcrash, which is the dump analysis tool that comes with the LKCD package. If we don't write the copy to a partition or other block device, we must modify the lcrash program to be able to read from regular files. In either case, the command to save the dump is:
# lcrash -P -s dumpdev dumpdir
where dumpdev is the partition or file containing the raw dump, and dumpdir is the directory in which the dump should be saved. The dump will be saved as a filed named "dump.0". Analysis with Crash The "crash" utility is another great tool for analyzing crash dumps, and it understands the dump format used by LKCD. It uses the GNU debugger (gdb), which is provided in the package, for most of its work. I'm using crash version 3.7-5 to analyze the LKCD dump that I got from the victim system. In the common case, the crash command takes two arguments. One is the kernel, which contains debugging symbols, and the other is the dump file generated by the crash dump facility. The debug kernel can be generated by adding the "-g" flag to the "CFLAGS" define in the top-level Makefile when building the Linux kernel. Crash will use this kernel to find the symbols it needs when analyzing a dump file. Make sure that the kernel on the victim system and the debug kernel are the same version and use the same source code. Otherwise, crash will complain that the versions don't match:
# crash vmlinux.dbg dump.0
crash 3.7-5
Copyright (C) 2002, 2003  Red Hat, Inc.
Copyright (C) 1998-2003  Hewlett-Packard Co
Copyright (C) 1999, 2002  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public 
License, and you are welcome to change it and/or distribute copies 
of it under certain conditions.  Enter "help copying" to see the 
This program has absolutely no warranty.  Enter "help warranty" 
for details.
GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, 
and you are welcome to change it and/or distribute copies of it 
under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" 
for details.
This GDB was configured as "i686-pc-linux-gnu"...

      KERNEL: /evidence/vmlinux         
    DUMPFILE: dump.0
        CPUS: 1
        DATE: Sun Nov  9 18:47:26 2003
      UPTIME: 00:33:39
LOAD AVERAGE: 0.00, 0.01, 0.00
       TASKS: 36
    NODENAME: host
     RELEASE: 2.4.20
     VERSION: #19 SMP Mon Nov 3 21:47:26 EST 2003
     MACHINE: i686  (1799 Mhz)
      MEMORY: 160 MB
       PANIC: "sysrq"
         PID: 0
     COMMAND: "swapper"
        TASK: c037a000  
         CPU: 0
When crash starts, you'll notice that it provides some interesting information about the system. Most notable for a forensic investigation is the current date "Sun Nov 9 18:47:26 2003", the uptime "00:33:39", and the release "2.4.20". The first two pieces of information will be useful when constructing an event timeline, and the release will be useful for finding the correct version of the kernel source code. The panic message "sysrq" also validates that the system dumped because the investigator instructed it to do so, as opposed to crashing for some other reason. At the crash prompt, you can use the "help" command to get information about the other commands available and their usage. As a first step, we can use the ps command in crash to find a list of processes that were running on the system:
crash> ps
   PID    PPID  CPU   TASK    ST  %MEM     VSZ    RSS  COMM
>     0      0   0  c037a000  RU   0.0       0      0  [swapper]
   1209   1112   0  c87ee000  IN   1.3    6796   2116  sshd
   1211   1209   0  c86d6000  IN   0.9    4312   1412  bash
   1266   1211   0  c8580000  IN   0.2    1360    248  malware
   1267   1211   0  c8550000  IN   0.2    1360    248  invisible
   1312   1211   0  c83ee000  IN   0.9    4668   1460  tcsh
Note that crash wasn't fooled by two of the hidden processes -- process 1266, which was obscured by the trojan ps, and process 1267, which was hidden by the adore LKM. Adore replaces the "getdents" system call so that when ps looks in "/proc" to get the list of processes, it doesn't find the invisible processes. Crash doesn't use the getdents system call; instead it reads the memory directly. Process 1268, which adore removed from the task list entirely, will be a little trickier to find. In crash, we can enter a kernel symbol name, and the symbol's value will be returned. In the case of a structure, crash will report the values of each element of the structure. The Linux kernel keeps hash lists of all the TCP sockets. The two lists of interest to us are the "established" list, named "__tcp_ehash", and the "listening" list, named "__tcp_listening_hash". These are both members of the "tcp_hashinfo" struct in the Linux 2.4 kernel. We can use these to find any TCP sockets that netstat didn't report:
crash> tcp_hashinfo
tcp_hashinfo = $4 = {
  __tcp_ehash = 0xc9f40000,
  __tcp_bhash = 0xc9f20000,
  __tcp_bhash_size = 16384,
  __tcp_ehash_size = 8192,
  __tcp_listening_hash = {0xc9e39500, 0xc9e39c00, 0x0, 0x0, 0x0, 
    0x0, 0x0, 0x0, 0xc86fcb00, 0xc86fc780, 0xc9e67180, 0x0, 0x0, 
    0x0, 0x0, 0xc9e38400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc9e39180, 
    0x0, 0x0, 0xc95e0480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0},
We can use the p command to print out the value of an expression and also use the * character to de-reference a pointer. When looking through each of the buckets in the listening hash, we find something interesting when we reach bucket 10:
crash> p *tcp_hashinfo.__tcp_listening_hash[10]
$13 = {
  num = 41002,
  sport = 10912,
  sleep = 0xc88c06bc,
This is a socket listening on TCP port 41002. Note that the source port ("sport") value of 10912 is the network byte order representation of 41002. This can be shown by swapping the bytes of the short from big-endian byte order to little-endian. We can find out which process is listening on this port by looking at the task_list pointed to by "sleep":
crash> p *tcp_hashinfo.__tcp_listening_hash[10].sleep
$16 = {
  lock = {
    lock = 1
  task_list = {
    next = 0xc8519e80,
    prev = 0xc8519e80
"sleep.task_list" contains a pointer into the stack of the process that is waiting on this socket. We know by examining the kernel source that each process is represented in Linux by a task structure that is at the bottom of two pages (8192 bytes) of memory. By "rounding down" this address by clearing the low-order 13 bits (2^13 = 8192), we get the address of the task structure. We can then have crash cast this address to a specific type and de-reference it by specifying the type, followed by the address. Also, simply specifying the type will show a list of elements in a struct, but specifying the type followed by the "-o" option will also show the offset of each element:
crash> task_struct c8518000
struct task_struct {
  state = 1,
  next_task = 0xc8548000,
  prev_task = 0xc8550000,
  pid = 1268,
  uid = 0,
  comm = "removed\000\000\000\000\000\000\000\000"
This task is in a "TASK_INTERRUPTIBLE" state. Note that the pointers to the previous and next tasks in the task list have not been cleared. These could point to other possibly hidden processes that are worth examining. In this case, one of the processes is 1267, which we already found, and the other already exited, freeing its memory for reuse. The pid is listed as 1268, which corresponds to a process we were missing from our ps listing earlier. We also see that the name of this process is "removed". Note that the entire comm buffer is printed, even though it contains a NUL-terminated string, which is why we see the escaped NUL characters. We've now located all of the processes that were missing. After examining all of the entries in the listening hash, we can go through the established hash. The kernel source will show that although the listening hash is of fixed length, we must look up the length of the established hash list. This value is stored conveniently in the tcp_hashinfo struct:
crash> p tcp_hashinfo->__tcp_ehash_size
$133 = 8192
It would take quite a long time to go through 8192 buckets by hand, so we can use the gdb backend to crash and automate this process by setting up a simple loop and printing out the entries that are in use:
crash> gdb set $i=0
crash> gdb while $i < 8192
 >if tcp_hashinfo.__tcp_ehash[$i].chain != 0
  >p $i
  >p tcp_hashinfo.__tcp_ehash[$i]
 >set $i=$i+1
This loop uses $i as a variable and increments it from 0 to 8191. If a chain of entries is attached to the hash bucket, the bucket number and the contents of the first entry will be printed out. The following will be printed in this case:
$19 = 296
$20 = {
  lock = {
    lock = 16777216
  chain = 0xc9e67500
Thus, bucket 296 has something in it. Let's examine it:
crash> p *tcp_hashinfo.__tcp_ehash[296].chain
$5 = {
  daddr = 17235978,  
  rcv_saddr = 2147942410,
  dport = 16256,   
  num = 22,
  sport = 5632,    
The four values of interest here are the destination address of this connection "daddr", the source address "rcv_saddr", the destination port "dport", and the source port "sport". These numbers may not look interesting because they are in network byte order, and also, the addresses are in decimal, which is difficult to read. We can use a built-in command to convert the addresses to something more familiar:
crash> net -n 17235978
crash> net -n 2147942410
We can then use a tool, such as the calculator "bc", to convert the port numbers from big-endian to little-endian. We then find a destination port of 32831 and a source port of 22. We can assume this is the server end of an ssh connection from the host at We can then examine the associated socket structure to find that the connection is in the connected, or established, state:
crash> p *tcp_hashinfo.__tcp_ehash[296].chain.socket
$19 = {
  state = SS_CONNECTED,
We can also find the owner of the socket by looking at the sleep structure:
crash> p *tcp_hashinfo.__tcp_ehash[296].chain.sleep
$65 = {
  lock = {
    lock = 1
  task_list = {
    next = 0xc840c02c,
    prev = 0xc840c02c
It contains a task_list, which is a pointer to the task_list in a wait_queue_t. By subtracting the offset of the task_list in the wait_queue_t from the address, we can see the full wait_queue_t. We can examine kernel header files for the wait_queue_t declaration and find that this number equals eight:
crash> p *(wait_queue_t *)0xc840c024
$68 = {
  flags = 0,
  task = 0xc87ee000,
  task_list = {
    next = 0xc879d8c0,
    prev = 0xc879d8c0
With the task_structure, we can find more information about the process; for example, we see that its pid is 1209 and its name is "sshd":
crash> p ((struct task_struct *)0xc87ee000)->pid
$70 = 1209
crash> p ((struct task_struct *)0xc87ee000)->comm
$135 = "sshd\000og\000\000\000\000\000\000\000\000"
We can also see a list of sockets used by each process by using the net command. The -s option to net will show us the sockets associated with a specific process. In the following case, we see that the two processes, 1266 and 1267, are listening on ports 41000 and 41001, which were missing from the netstat output:
crash> net -s 1266
PID: 1266   TASK: c8580000  CPU: 0    COMMAND: "malware"
 3  c88c0ea0  c86fcb00  INET:STREAM
crash> net -s 1267
PID: 1267   TASK: c8550000  CPU: 0    COMMAND: "invisible"
 3  c88c0aa0  c86fc780  INET:STREAM
By iterating through all of the buckets in the established and listening hash lists, we've now recovered all of the network sockets missing from the netstat output. Saving Memory with memget There are times when the kernel is built without any way to dump its memory to disk. For this, the investigator can use memget. Memget reads the sparse file "/dev/kmem" and transfers the data over a TCP network connection to the address and port specified on the command line for later analysis:
[[email protected]]# ./memget 1026
Saving memory...
Saved 45250 pages.
Using mempeek to Analyze Saved Memory Once the investigator has transferred the memory image to a forensic workstation, he or she can use mempeek to examine that image. This is a very crude way of examining the memory, especially compared to the interface of crash, but if a dump facility isn't available, it's better than nothing. For example, we know that the system time is stored in the xtime variable in memory. We can find the address of this variable in the file from the /boot directory:
% egrep xtime
c03de4b0 B xtime
It's a good idea to have the kernel source handy during the investigation. We can see that xtime is a "struct timeval" containing the number of seconds and microseconds since January 1st, 1970. Using the peek command at the mempeek prompt returns the requested address and the 32-bit data at that address represented in hexadecimal, decimal, and as a string of four characters:
mempeek> peek c03de4b0
c03de4b0: 0x3faed015 (1068421141) " P.?"
We find that the decimal value of xtime.tv_sec is 1068421141. This isn't very useful by itself, but we can use a Perl expression to convert it to a time in the local timezone:
% perl -e 'printf( "%s\n", scalar localtime(1068421141))'
Sun Nov  9 18:39:01 2003
This matches up pretty closely with the time that crash reported on startup in the previous investigation. The reason for the difference is that memget was run several minutes before the crash dump was taken. We can use mempeek to find the process hidden by adore. Adore sets two bits in the flags element of the task struct to mark a process as invisible. If no other flag bits are set, the value will be hexadecimal 0x03000000. To find the invisible process, we must first find the address of init_task_union.task, which points to the list of processes. We can obtain this address from /boot/ on the victim system:
% egrep init_task_union
c037a000 D init_task_union
We can then use mempeek to find the value at that address, as well as other values in the task_struct by specifying hexadecimal offsets after the address. We'll look for the flags, next_task, prev_task, and pid at offsets 0x04, 0x48, 0x4c, and 0x7c, respectively. If we were using crash, we could obtain these offsets by issuing the command task_struct -o. In this case, where the system isn't set up to take a crash dump, it probably doesn't have a debug kernel either so we'll have to determine the offsets by looking at the kernel header files. "next_task" will contain the address of the next process in the doubly linked list of processes, so we can use it to loop through the list:
mempeek> peek c037a000 4 48 4c 7c
c037a000: 0x00000000 (0) "    "
c037a004: 0x00000000 (0) "    "
c037a048: 0xc9d3c000 (-908869632) " @SI"
c037a04c: 0xc83ee000 (-935403520) " '>H"
c037a07c: 0x00000000 (0) "    "
mempeek> peek c9d3c000 4 48 4c 7c
c9d3c000: 0x00000001 (1) "    "
c9d3c004: 0x00000100 (256) "    "
c9d3c048: 0xc9d30000 (-908918784) "  SI"
c9d3c04c: 0xc037a000 (-1070096384) "  [email protected]"
c9d3c07c: 0x00000001 (1) "    "
mempeek> peek c8580000  4 48 4c 7c
c8580000: 0x00000001 (1) "    "
c8580004: 0x00000000 (0) "    "
c8580048: 0xc8550000 (-933953536) "  UH"
c858004c: 0xc86d6000 (-932356096) " 'mH"
c858007c: 0x000004f2 (1266) "r   "
mempeek> peek c8550000  4 48 4c 7c
c8550000: 0x00000001 (1) "    "
c8550004: 0x03000000 (50331648) "    "
c8550048: 0xc83ee000 (-935403520) " '>H"
c855004c: 0xc8580000 (-933756928) "  XH"
c855007c: 0x000004f3 (1267) "s   "
The last process has the flags value set to "0x03000000", which corresponds to adore's flags PF_INVISIBLE and PF_AUTH. So, we've now found process 1267, which was missing from the ps listing.


Sometimes data in memory on compromised systems, such as processes and network connections, are hidden from us by attackers. The attackers may have changed commands to hide this data or may have changed the behavior of the kernel to hide the data from various commands. In this article, I covered two tools that can be used to look at kernel memory. By examining certain kernel structures, I showed how to uncover the hidden processes and network sockets. Crash can be used along with a debug kernel to analyze the memory in a dump file symbolically. In the absence of a crash dump facility, memget can be used to save the system memory, and mempeek can be used to analyze the memory by address. Symbol addresses can be found in the /boot/ file on the victim system.


adore --

The Coroner's Toolkit --

crash --

ethereal --


memget/mempeek --

The Sleuth Kit --

snort --

tcpdump --

Michael Ford began his career as a network and systems administrator 10 years ago while attending the Massachusetts Institute of Technology. He holds the CISSP and GCFA professional certifications and is the founder of RND Software, Inc., where he performs security research and consulting. Michael can be reached at: [email protected].

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.