Saturday, 29 October 2016

WofCompressed streams in Windows 10

On windows 10, there is a new 'System Compression' option that compresses files using reparse points. This is not the NTFS-based compression that earlier versions of windows utilized, its different. This post is about the new compression scheme and how it affects forensic analysts.

With windows 10, a lot of details are automatically managed without user input and this is one of them. Windows can determine if the compression will be beneficial to the host system and automatically trigger it! This usually happens when you upgrade as opposed to clean installing the OS. Some users have reported seeing it as an option in 'Disk Cleanup' too.

Windows provides a utility called Compact.exe to do this processing manually. Using it, you can compress/decompress files and folders or simply query a system to determine if it will be beneficial at all on a specific volume. The compression algorithms are XPRESS (4K, 8K, 16K) or LZX. While the files are compressed on disk, if an application opens/reads such a file, it is still getting the original decompressed data and all decompression is handled on the fly automatically by windows 10.

Figure 1 - Compact.exe and its command usage info
The command 'compact /exe <file>' will compress any file (not just exe)

Lets get to the point, how does this impact forensics

Well, as of now, no tools will recognize and decompress these files. Hence, you can't read, keyword search or extract these files in their original uncompressed form.

Tools tested

Here is a list of tools tested so far:

Tool VersionSupport (as of 10/26/2016)
SIFT Workstation3No
Xways Forensic19.0No

How it works?

System compression utilizes reparse points and creates a new Alternate Data Stream (ADS) having the name 'WofCompressedData'. The compressed data is stored here. Reparse points are an NTFS feature that allow custom implementation like this. However this means that other applications that are not aware of this custom implementation will not be able to read/write to that file. In encase (or other forensic tools), you can see the file and the WofCompressedData stream. Clicking on the file just shows the contents to be all zeroes. Clicking on the stream, you can get the compressed data, but as of now, no automatic transparent decompression (as it does with NTFS compressed files). This is seen in screenshot below.

Note - This isn't to be confused with WOFF compression, which is a compression scheme used in Web Open Font Format!

Figure 2 - Encase shows the WofCompressedData stream. The file's original data was all text.
If you mount a volume containing such compressed files in SIFT Workstation or any linux system (they all use the same NTFS-3g FUSE driver), you will see the message 'Unsupported reparse point' when trying to list these files. Trying to access file contents will result in errors as seen in screenshot below.

Figure 3 - Files DW20.exe and upgrader_default.log are compressed here
If you attach a windows 10 formatted volume/disk to a Windows 7 system, you won't be able to access files as it does not know how to read them. See screenshot below:

Figure 3 - Notepad trying to view upgrader_default.log file (which is compressed)

Workarounds (till supported is added in by tool developers)

For Linux

If you use SIFT or another Linux system to do your forensics, the fix is simple. A few months back, Eric Biggers wrote a plugin to handle this. Its a plugin to the ntfs-3g FUSE driver. Its available here:

For this, you will first need to download, compile and install the latest version of the ntfs-3g driver (but not from Tuxera, that one is missing a file!); then proceed to download, compile and install the above mentioned plugin. You can get this working on SIFT with roughly the following steps:

1. Go to and download the source code for the latest stable release, right now its ntfs-3g_2016.2.22AR.1.orig.tar.gz.
2. Unzip and extract the file downloaded.
3. Open Terminal and browse to the extracted folder.
4. Compile and install using commands:
sudo make install
4. Go to and download the entire code as a zip file.
5. Unzip and extract the archive.
6. Open Terminal and browse to the extracted folder.
7. A few more tools need to be installed to compile this, so run the following commands:
sudo apt-get update
sudo apt-get install autoconf automake libtool
8. Run following commands to generate a configure script:
mkdir m4
autoreconf -i
9. Compile and install
sudo make install
10. If all went well (without errors), you are done!

Now you should be able to view and read those files normally, all decompression is handled on the fly automatically!

Figure 4 - No errors seen listing or reading files after installing the system compression plugin

For Windows

If you use Windows as your host machine for forensics processing, then you should only use a Windows 10 machine for processing evidence files that contain windows 10 images. This applies to tasks such as antivirus scanning, where you would typically share the entire disk out using Disk emulation (if you use Encase) which allow windows to parse and interpret the disk. This would only work (to read system compressed files) if the host system is Windows 10.

If you are looking to identify the system compressed files, you could filter on all files with ADS streams that have the name 'WofCompressedData'.

Fortunately, by default windows only compresses system files (EXE/DLL in windows and system32) and not user files, so you should mostly be fine. However, users can compress any file manually using the compact command.

Friday, 3 June 2016

Parsing the Windows 10 Notification database

Notifications on windows was a new feature added with windows 8 and continues in 10. In this post, I briefly discuss the format and data obtained from these notifications. Notifications can hold useful recent data (and some not so recent data) such as popup messages from applications, email snippets, application specific data like torrent downloaded messages among other information. As of now, not many applications use this feature on windows (when contrasted to apps on mac), but that is changing as more applications begin adding support for sending events to the Notifications Center/Bar.

As pointed out by Brent Muir here, this database is located at:

This Notifications database holds not just the popup notifications which the user sees briefly, but also  any updates to Tiles on the new windows start screen/start menu. Under the notifications scheme used by windows, there are 4 types of notifications, Toasts (popups), Tiles (updates on app live tiles like latest news stories, tweets or weather), Badges (small overlay on tile used to show status or count of items) or Raw push notifications (app specific data).

Appdb.db is a binary database having the signature 'DNPW' as the first 4 bytes. The structure of the file is roughly as shown below:

By default, there are 256 chunks in the file. Each chunk has a header element, however, only the first chunk has the header filled in. The chunk header starts with the DNPW signature, followed by what I believe to be the time the last notification was displayed to the user (8 bytes FILETIME) and the next sequential Notification ID to be used, and some unknown data after that (12 bytes).

The header is followed by data that I assume to be flags (8 bytes), followed then by Push URI (URL used by apps to push data and notifications to the client), Badge XML content and Tile Data (5 metadata objects and 5 corresponding XML data strings). Each of these elements in the chunk has its own data structure, which is quite detailed in itself. I am not reproducing all the structures here. To get this information, download the 010 Template (from link below) containing all the definitions for structures (deciphered so far..). There is also a python script available to parse information from this file and write out to a CSV file.

Tuesday, 10 May 2016

Amcache on Windows 7

The amcache registry hive which made its debut in windows 8, is now also showing up on Windows 7 systems. I was alerted to this by a fellow DFIR analyst Clint Hastings, who noticed this and has been using my scripts to parse them on windows 7 for some time now.

Amcache on Windows 7

So, what happened? After a bit of investigation on my machines, it was traced to Windows Update KB2952664, which updates the application inventory and telemetry (Microsoft terminology for the programs that monitor application usage) executables and libraries.

The update first came out in April 2015, but it appears as if it was not widely deployed (automatically) until around October.

Both Amcache.hve and RecentFileCache.bcf are updated now. I verified this information by parsing both these artifacts. Amcache of-course, had a lot more detail about the same files. So, don't forget to look for amcache on your windows 7 examinations.