2-UNIX Live Response

1. 2-UNIX Live Response John P. Abraham Professor University of Texas Pan American

2. Unix LIVE RESPONSE • Almost identical to windows • The only difference will be the commands that we’ll use to acquire the evidence. • We will be running the same netcat commands to transfer the live Response to the forensic workstation. • First, on the forensic workstation, we will run the following: • Nc –v –l –p 10000 > command.txt • Now, any data sent to TCP port 10,000 on our forensic workstation will be saved to command.txt. • On the victim computer, you will want to run a command to collect Live Response data. • The output of the command will be sent over our TCP channel on port 10,000 in this case. • Command | ncforensic_workstation_ip_address 10000 • After these commands have completed, you will press ctrl-c to break the netcat session, and our file command.txt will contain all the data from the command we executed. A simple MD5 checksum of command.txt can be calculated so that you may prove it s authenticity.

3. Analyzing Volatile Data • The System Date and Time • Date command • Current Network Connections • Netstat –an • We find an intrusion on Port 515 which is the port on which the printerdaemontypically listens. • We go to http://www.securityfocus.com to find that this is a vulnerable TCP port. This may have been the initial point of intrusion. • Open TCP or UDP Ports • Netstat –an command • Through examining the open ports, we hope to discover any backdoors the attacker may have established. • Netstat –anp will list the process number that opened the port also. This only works with Linux and will not work on UNIX

4. Executables opening TCP or UDP ports • IN UNIX we use lsof for list open files. It displays the processes that open ports, but also lists the files that the process has open. It is the single most powerful tool in our live response toolkit for UNIX systems. • Lsof –n produces lengthy output • The fourth bolded line on page 52 shows that file descriptor #2 (aka the standard out) is redirected to /dev/null. This indicates that the errors from this program will not be sent to the console but instead will be thrown away. • The next bolded line shows that the “printer” port is involved in an active connection for this process. • The last two lines show an open TCP port (3287) and an established connection. It would be pretty safe to say at this point that the IP address of 94.90.84.93 is probably the intruder’s origin. • Because we brought a trusted copy of lsof from an uncompromised machine, we have to surmise that the mechanism the intruder used to hide the open port must exist at the kernel level in addition; we will see that /tmp/.kde does not show up on the file system when we list the time and date stamps. That solidifies our theory that the kernel is probably trojaned, possibly with a loadable kernel module (LKM). A good resource: • Malware: Fighting Malicious Code

5. Continued.. • Running processes • To view the open processes: • Ps –aux • This command lists all the running processes on the system. And the users running them. • Open Files • Use • Lsof • To view open files • Once we run on our victim machine, we realize that the first process has a current working directory of /tmp/.kde. The second process has a current working directory of /tmp/.kde/brute/john-1.6/run. This is not good! Someone may be running John the Ripper on BRJDEF. John the Ripper is a common password cracking program that attackers employ to learn users’ passwords. • The Internal Routing Table • We obtain the routing table • Netstat –rn • In this case, the routing table looks intact.

6. Continued.. • Loaded Kernel Modules • Because we suspect that the kernel may have been trojaned, we will want to review all the loaded kernel modules. This is done with the lsmod command. When we dump the loaded kernel modules, we see the information on page 57. • There exist ways to hide a module after it is loaded. After a module is hidden there is no way (in the Live Response process) we can detect that it is installed. Furthermore, when we investigate Phrackmagazine, we see there are other ways to infect the kernel without even loading a module. • Mounted File Systems • The mounted file systems can be obtained from issuing either the mount command or the df command. • Output of the df command on bottom of page 57.

7. Analyzing Nonvolatile data • we would like to obtain several key pieces of information while the machine is still running. The type of data we will discuss in this section is nonvolatile. This means we could also retrieve this information from a forensic duplication if we so desired, but that option may be difficult or impossible to achieve. Some of the information we would like to acquire is as follows: • system version and patch level • file system time and date stamps • file system MD5 checksum values • Users concurrently logged on • A history of logins • Syslog logs • User accounts • User history files • Suspicious files

8. System Version and Patch Panel • When we issue the command uname –a we receive all the available operating system version information. • The patch level information for a Linux machine is a little more difficult to retrieve because it is dependent on the Linux distribution used to create the system. • In this case, it is RedHat 7.0 so we issue the rpm –qa command • File System Time and date stamps. • File system time and date stamps can be obtained using the same command for windows

9. Continued.. • The following command we can print out the file permissions, last access date, last access time, modification date, modification time, inode change date, inode change time, user ownership, group ownership, file size, and full path of the file for the compete file system: • Find / -printf “%m;%Ax;%AT; %Tx;%TT;%Cx;%CT;%U;%G;%s;%p\n” • We will sort all the files by the time the inode was last changed (also known as the “ctime”). We do this because there is no creation date on Unix file systems. The change date is as close as we can get to the creation time. After ruling out the other files changed on September 8, 2003, we have subset of files that we cannot explain. Pg 60. • It seems as if someone installed the Net::Telnet module. Perhaps this was a dependency to one of the tools that the intruder ran on our system. • We also see that the /etc/passwd/ and /etc/shadow were altered. This is usually done when an intruder adds a backdoor in the form of a valid user on the victim machine. • The other thing we see is two versions of lpd. The first instance in the table is probably the real lpd program.

10. File System MD5 Checksum Values • MD5 Checksum- is a 128-bit mathematical fingerprint of the contents in a file, for every file on the file system. • After we calculate the MD5s, we can compare them to databases of known MD5s. • www.hashkeeper.org • There is also the NSRL software reference library. These databases will help us eliminate the known files and leave us with only unknown files to examine. • To calculate the MD5 checksum for every file on the system, you would issue the following command: • Find / -type f –xdev –exec md5sum –b {} \; • The results for this sweep are on the DVD.

11. Users currently logged on • The users who are currently logged on are saved in the /var/run/utmp log. We could reconstruct this information from a forensic duplication, but it is much easier to acquire this information with the w command. Output on page 62. • We see that root, Curtis, and lpd are all logged in. the most important line that should be evident her is the lpd loginlp is a valid system account, and perhaps the attacker hooped that lpd would blend in. • We also see that the attacker’s IP address is 94.90.84.93. • Even though these logs are in binary format, the format is well known. If the attacker chose to run a program called zap2 he could have easily zapped the lpd login that we see here.

12. A History of logins • A history of logins is saved in the /var/log/wtmp binary log, so we can reconstruct this information form a forensic duplication. We can view this information easily during our Live Response. • Syslog logs • Unix systems have a syslog facility • The syslog daemon listens for messages from either local programs or other servers on the Internet and logs them according to the /etc/syslog.conf configuration file. • The only logs that will be relevant to our investigation are /var/log/messages and /var/log/secure. • Using netcat we transfer relevant logs to our forensic workstation for further analysis. • In the messages log, we see that on September 8, at approximately 14:35, a series of events begins that does not make much sense (and that is difficult to even copy to this page). The message looks like binary data written to the log. This is typically an indication of a buffer overflow attack. When buffer overflows occur, they break valid programs. When programs break, garbage is typically generated in the log. In this scenario, we cannot see what program was attached. Other overflows will report the name of the service that wrote to the log, such as the rpc.statd buffer overflow, which reports that rpc.statd wrote to the log. • On the log we notice that kjones failed at login and then successfully log in as matt shortly after. Dead end as explained on top of page 66.

13. User Accounts (page 66) • We know the attacker is creating backdoors on BRJDEV at this point. Let us examine the /etc/passwd file to see whether the intruder has added any rogue user accounts. • Once we see it, the normal system and user accounts make up every line except the last. We see that there e is a new account, called lpd. It has a root directory of/ and, more importantly, a user ID of zero. This means that when lpd logs in, he will have root privileges, even though he will have a different password than root. • User History Files • History files are commands the user typed at the prompt. The files may contain commands that failed, or we may discover the hacker’s methodology. The one account we know the attacker used was the lpd account. When we examine lpd’s home directory (/), we do not find a history file. We also know that the attacker used the Richard account because it initiated the investigation. We do, however, find a bsh history file for Richard. Pg 67. The bolded lines show activity that Richard did not recognize.

14. Suspicious Files • If we were not acquiring a forensic duplication of BRJDEV, we could transfer any suspicions file with our netcat. • Nc –v –l –p 10000 > filename • Cat filename | ncforensic_workstation_ip_adress 10000 • Because BRJDEV’s kernel was probably trojaned, we would collect any of the suspicions files on the file system through the forensic duplication process to make sure we have a pristine copy of the tools. • There is a way we can acquire copies of running processes. In windows, an executable cannot be deleted while it is running in memory. The operating system locks the file and it cannot be removed. That is a benefit for us during an investigation. In Unix, an attacker can run a file, such as datapipe, and delete the original binary. If we were to shut down the machine in a graceful manner, we would not have a copy of this file for analysis later. This is where the /proc file system comes into play • The /proc file system does not actually exist on the hard drive. It exists in memory and references running processes and other system information. By entering the /proc directory, you see directories named after integers, such as 1348.