Tuesday, October 15, 2019

Part 2 - ADB keyvalue backups - Call Logs

This is Part 2 of the continuing blog series on ADB keyvalue backups. Today we focus on Call Log Backups. 

Call logs are backed up under 
They are backed up only if you specified the -keyvalue option and are available on non-rooted devices too.

This file follows the Key-Value Data format as outlined earlier in part 1. The Keys here are the call ids or serial number of calls, starting at 1 and sequentially rising. The Values are the individual call log records.

Here are the structures used in the Call Log record. All fields here are stored as Big Endian.

1. Text_Record

field_length (in bytes)
field data (text)

2. Call_Log

version, 0x03EF (1007) or 1005 seen
call duration in seconds
present if is_phone_number_present = 1
call type
   1 = Incoming
   2 = Outgoing
   3 = Missed
   4 = voicemail
   5 = Rejected / Declined
   6 = Blocked
   7 = Answered_Externally
number presentation
  1 = Allowed
  2 = Restricted
  3 = Unknown
  4 = Payphone
present if is_ servicename_present = 1
present if is_ iccid _present = 1
present if is_ iccid _present = 1
unknown bytes, always 0
oem namespace string
unknown bytes
block reason (only on version 1007)
   1 = Screening service
   2 = Direct to voicemail
   3 = Blocked number
   4 = Unknown number
   5 = Restricted number
   6 = Payphone
   7 = Not in contacts
unknown bytes (only on version 1007)

The screenshot below shows a raw record in the hex editor.

Figure 1 - Call log record showing some important fields

Using an 010 template to parse this information, it looks like this (below).

Figure 2 - Call log record data parsed in 010 editor
The level of detail on these records is great. There are call status codes known as Call Type (Missed, Incoming, Outgoing, ..) as well as a number Presentation code which is usually 1 (Allowed), although there are a few other values. Calls that show up on your phone as 'Private' numbers, will have presentation code 2 (Restricted). If you have enabled any call blocking features, then those show up too on blocked calls (known as block reason).

Code to automate this parsing

A python script has been created to parse call log records from the com.android.calllogbackup.data file, available here. The 010 template can be downloaded here.

Forensic Gems - Detecting Deleted call records

Since each call record has a key which is the call id or the serial number of the call, I performed an experiment to see if deleting intermittent call records would change this number. It turns out that the number does not change, in effect allowing us to detect deleted call records. This is visible in the screenshot below, where you can see call IDs (serial numbers) of 1 through 8 but its missing 4 and 7. Those are the ones I had manually deleted from the Call logs on the phone through the available feature in the Phone app. This was also tested on a real phone with several hundred call records going back several months and it appears to hold true there too.

Figure 3 - Output of callparser.py, made pretty in Excel showing missing call ids.
This can be useful knowing that there can be records which are missing, perhaps intentionally.

Stay tuned for Part 3, there is more good stuff in these key-value backups.

Monday, October 7, 2019

ADB keyvalue backups and the .data format

The ADB backup has been a very useful tool for getting data from Android phones, particularly those phones/devices that are otherwise not accessible due to lack of support by forensic software vendors or hardware/software issues with other methods.

There is however one feature which I do not see being used by any of the vendors or FOSS or any other guides out there. I am specifically talking about about ADB's backup feature to backup key-value pairs. According to one source, since Oreo (8.0), the keyvalue backups are now available via adb backup.

To get data with keyvalue pairs returned, you need to add the -keyvalue parameter to the adb backup command like. I like to use :

adb backup -all -shared -system -keyvalue -f file.adb

Keyvalue backups give some very good information otherwise not available in the adb backup.

So, where are the key-value backups located?

When viewing the adb tar archive, you will find one or more folders under each app's folder with names like k, sp, db, .. The k folder holds the key value backup, having file(s) which ends in the extension .data.
Figure 1 - Folders holding key-value .data files (this isn't all , there are many more)
The *.data files are located in the k folders usually having the same name as the package like com.android.calendar.data.

Parsing .data files

This consists of a series of records, each starting with 'Data', and having a key (name) and value (data). The format is as follows, all data in this structure is stored as little-endian:

12 + key_size
pad to 4 byte boundary
pad to 4 byte boundary

The value field can be different types depending on the data/database being backed up. It is different for different packages. You can find XML files, entire SQLITE databases in there, and also single byte true/false type settings.

In the screenshot below, you can see the key-value records as parsed out for com.android.vending.data. The 010 template for this is available here.

Figure 2 - Hex editor view of com.android.vending.data, showing 'Data' records parsed out using an 010 template

In the above example, the value types are mostly True/False. But most other databases have other custom structures embedded there, which need further parsing.

In part 2 of this ADB series, we explore the formats of call logs and other databases that are backed up. 

Saturday, December 29, 2018

Making NSKeyedArchives human readable

If you've been doing macOS analysis, you are definitely familiar with the (now not so new) serialized plist format also known as an NSKeyedArchive. There are parsers available to extract data from this format, such as the ccl_bplist from Alex Caithness (some more sophisticated ones listed at the bottom of this post). I've been using this library in mac_apt and other projects too. That is all old news.

Need for a human readable version

I always like to manually verify the results of my code by looking at the raw plist values to make sure my analysis programs parsed the right values as well as if they missed something. That isn't possible with NSKeyedArchives.

The ccl_bplist parser is great if you want to explore the structure of a plist interactively with python or programatically. What it does not do is auto-generate and save the deserialized version of that plist as a new human readable plist file. So that is what I set out to do yesterday. 

Long story short, the code is available on github here

Below are screenshots showing an NSKeyedArchive sfl2 file (Figure 1) and its deserialized human readable form (Figure 2).
Figure 1 - NSKeyedArchive

Figure 2 - Deserialized form of NSKeyedArchive

Using ccl_bplist and biplist, this should have been a 2-5 line program, ccl_bplist to deserialize and biplist to write out the new plist. However it turned out to be a few lines more than that as I had to write a short recursion function to process the plist data because the ccl_bplist also includes $class information which needed to be stripped out. Also as I found out, this will only work with Python 3 because Python 2 does not distinguish between the types str and bytes. In Python 2, all these (below) are the same:

This creates a problem as biplist cannot distinguish between binary blobs and strings as they all appear to be strings and it fails when it encounters a byte that cannot be encoded as a string. No problems with python 3 as these are different distinct types there.

The code is here.


Since writing this, I did find another library (bpylist2) that has similar functions, reading/writing binary plists as well as creating/reading keyed archives. I haven't tested it yet.

(Update) Our good friends at Aon also maintain a library for plist parsing which seems pretty solid and worth checking out here.

Saturday, November 24, 2018

$Recycle bin and Undo operations

This week Phil Moore made an excellent finding (link here), one that most of us have seen for years but not investigated. Those $I files that seem orphaned/abandoned without explanation now have one. Phil notes that every time a file/folder is deleted and then restored, the $I file stays behind in the recycle bin. This post is about some follow up testing and results.

I did some quick tests - deleting and restoring files in a few different ways testing $I & $R file creation every time. I'm not even looking at indexes or timestamps, just file creation/deletion. My testing was on Windows 10 (32bit) version 10.0.10586.106

I'm not reproducing all the output here, but only the summary. Here are a few ways to send files to the recycle bin. Remember you cannot do any operations using the command line (as that operates directly at the file system level and does not use the recycle bin abstraction).

First we delete a file (right-click and select Delete). Let's try to restore now using any one of the following ways:

1. Right-click the file, click Restore on file in the recycle bin
2. Cut file from recycle bin and Paste elsewhere
3. Drag the file from recycle bin and into another folder
4. Undo the last operation using Ctrl-Z  OR  right-click & select Undo Delete

Only the 4th method (Undo Delete) results in deletion of the $I file. The other methods leave it behind.

I believe this to be a windows bug. Interestingly, when you Restore a file from the bin (right-click & Restore), then right-click on the desktop (or in any folder), the context menu has an new item called 'Undo Move'. It does NOT say 'Redo Delete'. For every other action, you will see a 'Redo ACTION' in the menu (see figure 2). So I believe when you restore a file, windows just performs a file move (on the $R file) and thus marked it as a move in the last performed action. 

Figure 1 - 'Undo Move' seen after restoring file
Now, clicking on the Undo Move will result in sending the file back to the recycle bin but as a new $I & $R pair, as if it was an entirely new delete operation. 

Figure 2 - 'Redo Rename' seen after using 'Undo Rename' from an unrelated file rename operation
In figure 3 below, you can see multiple abandoned $I files after doing this multiple times (delete, then restore).

Figure 3 - Recycle bin folder after multiple deletes and restores showing abandoned $I files
Let us know (comments or twitter) if you know of more ways to delete/restore.

Tuesday, November 6, 2018

The ._ (dot-underscore) file format

If you've ever looked at removable media and found several hidden files which start with ._ and there exists one for almost every file (or folder) on the disk, this is the result of having that media being used on macOS.

macOS keeps a copy of file metadata in a separate area, known as Extended Attributes (xattr) on HFS+ or APFS. However, when writing to external media which is not formatted as HFS+ or APFS (thus not having the capability to store extended attributes), it will write this information out as a separate file which will have the same name, just prefixed with dot-underscore  ._  as seen in the screenshot below.

Figure 1 - Screenshot showing exFAT volume on External USB disk
While this is well known for many years, this information is often overlooked in a forensic investigation. On media that has interacted with both macOS and Windows (or even Linux), macOS will create these files and delete them too when the original file is deleted. However, if the file is deleted or renamed on Windows or Linux, then the dot-underscore files will be left behind untouched. A while back, Lee Whitfield touched upon this here specifically pointing out its use for knowing the date & time that a file was copied onto the media. However, there is useful information inside the file too. 

This file can contain useful metadata such as kMDItemWhereFroms (URL of file if downloaded from internet) and kMDItemDownloadedDate (Date & Time when it was downloaded) among other extended attributes.

After a bit of reverse engineering, I wrote an 010 hex editor template to parse this information, it is  available at https://github.com/ydkhatri/MacForensics/blob/master/DotUnderscore_macos.bt

In the screenshot below, you can see it being run on one such file. 

Figure 2 - DotUnderscore_macOS.bt template output

Here is analysis from the data extracted:

Attribute Name Value Meaning
kMDItemWhereFroms https://upload.wikimedia.org/wikipedia/commons/3/3c/Thiruvalluvar_Statue_at_Kanyakumari_beach.jpg URL from where it was downloaded
kMDItemDownloadedDate 0x41BFB51D1CFFA4F8 (11/09/2017 23:32:44) Timestamp when file was downloaded
com.apple.quarantine 0083;5a04e59c;Safari;A451620D-2B49-49BD-ADC1-88DEBEA66582 File was downloaded using the Safari browser*

kMDItemDownloadedDate is stored in a plist as a date value, which is stored as a 64-bit double that is the number of seconds since 01/01/2001
The template does not parse the plist for you, you can export it out and open in any plist viewer to view the human-readable date value.

*The 3rd item in com.apple.quarantine's value (separated by ;) is the Application (agent) name which downloads the file. For more details on this, read Howard Oakley's blog post