Getting some terms right:

Malware - malicious software - software that can cause harm to a user, computer or network in different ways such as taking the business down, hiding files, stealing info etc.

Malware Analysis - process of dissecting malware with the intention of finding out how it works, how to detect, defeat, remove and recover from it.

Signatures: patterns used to detect malware.

Host based signatures :

These are indicators used to identify the presence of the malware on a victim’s computer. The indicators can be files created or modified by the malware, registries and so on.

Network signatures

These are indicators used to detect malicious code by monitoring network traffic. The indicators can be specifically crafted packets, patterns in ports opened/accessed, patterns external URLs and IP addresses contacted, files downloaded and so on.

Analysis Techniques: Common techniques used to analyze a malware

Static Analysis

This involves examining the malware without actually executing it. ○ Basic static analysis - looking at the file, its characteristics, digital signature, resource sections, imported and exported functions etc. ○ Advanced static analysis - this involves digging deeper into the assembly level instructions in a disassembler and studying the code flow.

Dynamic Analysis

This involves running the malware and examining its behavior. (ie actually executing it) ○ Basic dynamic analysis - involves just running the malware in a protected environment to learn assess its behavior and possibly generate signatures. There are chances you may miss out on several aspects of the behavior. ○ Advanced dynamic analysis - involves running the malware under a debugger and studying its internal state at various points to extract detailed information of its behavior.

Hybrid Analysis - mixture of both!

Types of Malware:

When analyzing malware it is important to make an educated guess into what the malware is doing and then classifying it based on its behavior. Obviously there are others, and many a times a malware may fall into multiple categories at the same time. Some of the common categories of classification are:

Backdoors : Malicious code that installs itself on the victim’s computer and usually allows an attacker to connect to the computer with little or no authentication and execute commands.

Rootkits : Malicious code that is designed to conceal its own existence, or existence of other malicious code.

Trojans : Genuine looking software, with malicious code packaged into it.

Viruses and worms : Malicious code that can self-replicate and infect other computers over the network. Viruses are file based, ie replicate using a file, while worms typically use exploits and don’t use files to replicate.

Scareware : Malware that is designed to frighten the user into buying something,

Botnet : It is similar to a backdoor in that it allows attackers to access the system. The key difference is that botnets typically receive instructions from a Common and Control (C&C) server instead of an attacker manually accessing it.

Downloaders : Malicious code that downloads and installs other malicious code.

Launchers : Malicious code that is used to launch other malicious code in order to ensure stealth or gain greater access to the system.

Spyware : Gathers information from victim’s computer without his knowledge and sends it to attacker. Ex: keyloggers

Spam sending malware : Gains access to user’s computer and then uses it to send spam to other users.

Adware : hogs the victim’s computer with unwanted ads.

Detection techniques:

Antivirus programs are able to detect most known malware by using different detection techniques and signatures. Older AVs relied on a database of file signature, but this is not very reliant. The malware code can be easily re-arranged and re-compiled to change its signature. Malware authors may also use packaging and obfuscation to make their files more difficult to analyze and detect. Newer AVs rely on behavioral and pattern matching heuristics.

When chasing a malware, here are some of the things that I keep in mind:

Ports – what ports are being used, is it a common number pattern etc.

List of running processes – are there anything unexpected, tools like process explorer can visually indicate unsigned binaries

Registry – any new/modified settings

Drivers – is any unauthorized driver loaded and running?

Services – is any unauthorized service loaded and running?

Startup locations – is any suspicious executable being forced to startup. This may include RunOnce entries, scheduled tasks, file explorer extensions etc. Autoruns tool from sysinternals is a good one to use.

Files and folders – anything matching known patterns

Network activity – anything that matches baseline known patterns

AV scanners – if you have a suspect binary, using a service such as virustotal will help you eliminate known patterns.

Practical malware analysis requires sound knowledge of Assembly Language and Processor Internals, Operating Systems, if you work on Windows like me, reading Windows Internals book is a must do on the list. I have a planned series of blogs on basic assembly - let’s see how that goes, then I will have some on Windows Internals too.

Good books for reference:

Here are a few books I always have for references and reading. The stuff in there can get very involving and its difficult if you are in the midst of disturbance. I prefer to read them only when I know I will not be disturbed for the next 5 hours :)

Operating Systems by Galvin and Gagne. This is often a syllabus book for under grad OS class, and in my college days was it was known as the Dinosaurs book.
Windows Internals - now in its 6th edition - by Mark Russinovich and David Solomon
Cryptography and Network Security by William Stallings - is a favorite of mine, it explains concepts with well-crafted diagrams but is not very practical.
Practical Reverse Engineering by Bruce Dang and Alexander Gazette is a good read.
Advanced Windows debugging by Mario Hewardt, and Inside Windows debugging by Tarik Soulami covers windbg and other debugging tools in good detail.
Art of memory forensics covers the volatility framework - the backbone of Rekall.
Practical Malware Analysis - by Michael Sikorski and Andrew Hong. Is a good read and was recommended to me during an Interview.
Malware Analysts cookbook by Michael Hale and Steven Adair - has recipes to tackle common malware.