Kernel Hacking & Anti-forensics: Evading Memory Analysis

Dec 1 2009 in Articles by articles |

This article is intended to explain, why a forensic analysis in a live system may not be recommended and why the image of that system can trigger an advanced anti-forensic-capable rootkit.
Authors: RODRIGO RUBIRA, BRANCO (BSDAEMON), FILIPE ALCARDE BALESTRA
Source: Hakin9 http://hakin9.org

WHAT YOU WILL LEARN…

With this article you will better understand how the a computer arquitecture works and is closely related to the operating systems, focusing in subvertion of the memory acquisition process
Internal structures used to manage the memory, filesystem and others will be explained, using as sample the linux operating system, but trying to be generic enough to give a good idea of how it works in any platform

WHAT YOU SHOULD KNOW

In order to completely understand this article the reader must know about the Linux Kernel basic programming (how to create modules, how the basic kernel programming works) and also some of assembly and C language
Architecture internals will be well explained, but some computer science or engineering experience is required in order to have a real understanding of what is going on in the samples

Since, most of the operating systems have the same approach in this regard, most examples covered here in Linux can be applied to similar situations in other operating systems. An overview of the kernel internals and the structure and working of x86 architecture will also be given, along with the differences between other architectures.

Introduction

A lot of tools [5] have been developed to analyze a live system in order to detect an intrusion (like installed rootkits [7]). This article tries to explain some presentations [8] that showed problems in this existent model, explaining the risks of this act and when can it be accepted. Basics The chosen architecture was Intel x86, where the same concepts can be applied to other architectures as well(major modifications are needed in architectures without MMU). To better understand the following sections, some basic concepts are needed:

CPL0 and it is importance
System calls
Structures analyzed to memory management
Hook of functions and information flow

CPL0 and Its Importance

The Intel architecture has many levels of priority and the modern operating systems (Linux/ Windows/MacOS) are using that separation to provide protection and isolation of each process (so, a process cannot interfere in the execution of another one, or in the execution of the operating system itself). The operating system is executed in the CPL0 (also known as kernel-mode or ring0) because, in that mode any privileged operation is allowed (memory access, hardware management, and others). In this article micro-kernel operating systems are being ignored to facilitate the learning process. It is important to understand that the user applications are running in CPL3 (user-mode or ring3).

System Calls

When an usermode software needs some privileged resources (for example, read diskdata) it executes a system call. This is a software interrupt that turns the system into kernelmode, executing the system call handler to answer that call and then return the control to the usermode program. The way that system calls are handled is completely architecture-dependent. The common factor is that every implementation has similar structures, using different methods, libraries and other resources. In the following we discuss how this works in a x86 architecture (using int $0×80 instruction and the new way using sysenter).