Tuesday, October 15, 2019

Part 2 - ADB keyvalue backups - Call Logs

This is Part 2 of the continuing blog series on ADB keyvalue backups. Today we focus on Call Log Backups. 

Call logs are backed up under 
<Backup.adb>/apps/com.android.calllogbackup/k/com.android.calllogbackup.data
They are backed up only if you specified the -keyvalue option and are available on non-rooted devices too.

This file follows the Key-Value Data format as outlined earlier in part 1. The Keys here are the call ids or serial number of calls, starting at 1 and sequentially rising. The Values are the individual call log records.

Here are the structures used in the Call Log record. All fields here are stored as Big Endian.

1. Text_Record

Position
Type
Description
00
ushort
field_length (in bytes)
02
char[field_length]
field data (text)

2. Call_Log

Position
Type
Description
00
uint
version, 0x03EF (1007) or 1005 seen
04
int64
timestamp
12
uint64
call duration in seconds
20
byte
is_phone_number_present
21
Text_Record
present if is_phone_number_present = 1
..
uint
call type
   1 = Incoming
   2 = Outgoing
   3 = Missed
   4 = voicemail
   5 = Rejected / Declined
   6 = Blocked
   7 = Answered_Externally
..
uint
number presentation
  1 = Allowed
  2 = Restricted
  3 = Unknown
  4 = Payphone
..
byte
is_servicename_present
..
Text_Record
present if is_ servicename_present = 1
..
byte
is_iccid_present
..
Text_Record
present if is_ iccid _present = 1
..
byte
is_own_num_present
..
Text_Record
present if is_ iccid _present = 1
..
byte[12]
unknown bytes, always 0
..
Text_Record
oem namespace string
..
byte[18]
unknown bytes
..
uint
block reason (only on version 1007)
   1 = Screening service
   2 = Direct to voicemail
   3 = Blocked number
   4 = Unknown number
   5 = Restricted number
   6 = Payphone
   7 = Not in contacts
..
byte[18]
unknown bytes (only on version 1007)

The screenshot below shows a raw record in the hex editor.

Figure 1 - Call log record showing some important fields

Using an 010 template to parse this information, it looks like this (below).

Figure 2 - Call log record data parsed in 010 editor
The level of detail on these records is great. There are call status codes known as Call Type (Missed, Incoming, Outgoing, ..) as well as a number Presentation code which is usually 1 (Allowed), although there are a few other values. Calls that show up on your phone as 'Private' numbers, will have presentation code 2 (Restricted). If you have enabled any call blocking features, then those show up too on blocked calls (known as block reason).

Code to automate this parsing

A python script has been created to parse call log records from the com.android.calllogbackup.data file, available here. The 010 template can be downloaded here.

Forensic Gems - Detecting Deleted call records

Since each call record has a key which is the call id or the serial number of the call, I performed an experiment to see if deleting intermittent call records would change this number. It turns out that the number does not change, in effect allowing us to detect deleted call records. This is visible in the screenshot below, where you can see call IDs (serial numbers) of 1 through 8 but its missing 4 and 7. Those are the ones I had manually deleted from the Call logs on the phone through the available feature in the Phone app. This was also tested on a real phone with several hundred call records going back several months and it appears to hold true there too.

Figure 3 - Output of callparser.py, made pretty in Excel showing missing call ids.
This can be useful knowing that there can be records which are missing, perhaps intentionally.

Stay tuned for Part 3, there is more good stuff in these key-value backups.

Monday, October 7, 2019

ADB keyvalue backups and the .data format

The ADB backup has been a very useful tool for getting data from Android phones, particularly those phones/devices that are otherwise not accessible due to lack of support by forensic software vendors or hardware/software issues with other methods.

There is however one feature which I do not see being used by any of the vendors or FOSS or any other guides out there. I am specifically talking about about ADB's backup feature to backup key-value pairs. According to one source, since Oreo (8.0), the keyvalue backups are now available via adb backup.

To get data with keyvalue pairs returned, you need to add the -keyvalue parameter to the adb backup command like. I like to use :

adb backup -all -shared -system -keyvalue -f file.adb

Keyvalue backups give some very good information otherwise not available in the adb backup.

So, where are the key-value backups located?


When viewing the adb tar archive, you will find one or more folders under each app's folder with names like k, sp, db, .. The k folder holds the key value backup, having file(s) which ends in the extension .data.
Figure 1 - Folders holding key-value .data files (this isn't all , there are many more)
The *.data files are located in the k folders usually having the same name as the package like com.android.calendar.data.

Parsing .data files

This consists of a series of records, each starting with 'Data', and having a key (name) and value (data). The format is as follows, all data in this structure is stored as little-endian:

Position
Type
Description
00
char[4]
‘Data’
04
uint
key_size
08
uint
data_size
12
char[key_size+1]
key_name
12 + key_size
char[]
pad to 4 byte boundary
..
char[data_size]
data
..
char[]
pad to 4 byte boundary


The value field can be different types depending on the data/database being backed up. It is different for different packages. You can find XML files, entire SQLITE databases in there, and also single byte true/false type settings.

In the screenshot below, you can see the key-value records as parsed out for com.android.vending.data. The 010 template for this is available here.

Figure 2 - Hex editor view of com.android.vending.data, showing 'Data' records parsed out using an 010 template

In the above example, the value types are mostly True/False. But most other databases have other custom structures embedded there, which need further parsing.

In part 2 of this ADB series, we explore the formats of call logs and other databases that are backed up.