Monday, October 28, 2019

macOS 10.15 Volumes & Firmlink magic

With macOS 10.15 - Catalina, Apple has introduced a change in the way system and user data is stored on disk. In prior versions, the root '/' volume was stored in a single volume usually named 'Macintosh HD'. This did not change with the update to APFS. However with Catalina, there are now two distinct volumes -
  • Macintosh HD
  • Macintosh HD - Data
The screenshot below shows the two different volumes -

Figure 1 - diskutil output showing a split Macintosh HD volume into two

The Macintosh HD volume stores the system files and is mounted as read-only, while the Macintosh HD - Data volume has all the other files on your system which include user profiles, system and user data, and user installed Applications.

However when booted, only a single logical volume is presented (as root /) that combines the contents of both. This is enabled through APFS using its Volume Role feature. This is mentioned in Apple's official APFS documentation but its usage or working is not documented. Each volume can be assigned a role in its volume Superblock structure (apfs_superblock_t). There are 8 possible roles documented. From the Apple docs:

  #define APFS_VOL_ROLE_NONE      0x0000
  #define APFS_VOL_ROLE_SYSTEM    0x0001
  #define APFS_VOL_ROLE_USER      0x0002
  #define APFS_VOL_ROLE_RECOVERY  0x0004
  #define APFS_VOL_ROLE_VM        0x0008
  #define APFS_VOL_ROLE_PREBOOT   0x0010
  #define APFS_VOL_ROLE_INSTALLER 0x0020
  #define APFS_VOL_ROLE_DATA      0x0040
  #define APFS_VOL_ROLE_BASEBAND  0x0080

The SYSTEM volume contains the folders /bin, /sbin and most of the /usr and /System folders. A few subfolders of /usr and /System are on the DATA volume. The volumes are joined using a new construct that Apple calls firmlinks. They describe it as a Bi-directional wormhole in path traversal. Firmlinks are used on the system volume to point to the user data on the data volume.

They are somewhat similar to the unix symlinks and hardlinks, but only directories can be linked (from one volume to another). The file that defines/lists the firmlinks resides on the SYSTEM volume at /usr/share/firmlinks. The following paths are defined by default.

  /AppleInternal AppleInternal
  /Applications Applications
  /Library Library
  /System/Library/Caches System/Library/Caches
  /System/Library/Assets System/Library/Assets
  /System/Library/PreinstalledAssets System/Library/PreinstalledAssets
  /System/Library/AssetsV2 System/Library/AssetsV2
  /System/Library/PreinstalledAssetsV2 System/Library/PreinstalledAssetsV2
  /System/Library/CoreServices/CoreTypes.bundle/Contents/Library  
              System/Library/CoreServices/CoreTypes.bundle/Contents/Library
  /System/Library/Speech System/Library/Speech
  /Users Users
  /Volumes Volumes
  /cores cores
  /opt opt
  /private private
  /usr/local usr/local
  /usr/libexec/cups usr/libexec/cups
  /usr/share/snmp usr/share/snmp

The linked volumes will have distinct inode numbers for files/folders. The only common inode numbers seen are inodes 1 (Parent of root), 2 (root) and 3 (private-dir). All other inodes will be unique, a simple but clever scheme is used to ensure that. For the SYSTEM volume, every inode number allocated will be OR'd with 0x0FFFFFFF00000000. Take a look at the inode numbers in the  combined volume in screenshot below. The very large numbers are the files that reside on the SYSTEM volume due to the upper bits being set by the mask.

Figure 2 - Contents of root showing files from both SYSTEM and DATA
If you try to create a file or folder on the root volume (or one of its owned folders), it fails with an error: Read-only volume.

For accessing most files and folders, there should be no problem as the stitched/combined volume works seamlessly so all programs should not notice any difference. However there are situations where you might want to explicitly access a folder from a specific volume, especially for forensics. For example, if you wanted to access /./fseventsd, you would always get the read-only volume's .fseventsd folder which won't be too interesting as its a read-only volume! To get the one on the DATA volume, there is still a way. Apple has also made the DATA volume available (mounted) at the mountpoint /System/Volumes/Data. This also means that if you have scripts that run across all files, they will need to be made aware that this location should be avoided to prevent duplication.

According to Apple, you cannot opt-out of this, and it is a required feature for macOS 10.15. Forensic tools that operate on full disk images will have to adapt for this change, and so I've updated mac_apt to support macOS Catalina. If you use it, let me know of bugs/issues.

Saturday, October 26, 2019

Part 3 - ADB keyvalue backups - Wifi and System settings

This is Part 3 of the continuing blog series on ADB keyvalue backups. Today we focus on Wifi settings and other system configuration available from-
<Backup.adb>/apps/com.android.providers.settings/k/com.android.providers.settings.data
They are backed up only if you specified the -keyvalue option and are available on non-rooted devices too.

This file follows the Key-Value Data format as outlined earlier in part 1. There are 8 different types of data seen here. The Key name represents the type of data and Value represents either a single structure or a set of name-value pairs (both name and value are strings). The table below shows the data seen here.

Key Name
Description
system
settings about font sizes, screen brightness, hearing aids, haptic feedback among others
secure
more system settings on gestures, button behaviors, spell checker, screensaver, accessibility, etc..
global
Boolean settings that enable/disable options like wifi wakeup,   auto_time, sounds enabled, call auto-retry, etc..
locale
a locale string like ‘en-US’
lock_settings
owner info for display on screen if enabled
softap_config
Access point settings for Mobile hotspot
network_policies
unknown
wifi_new_config
xml data having wifi settings for connected access points

Perhaps the most interesting aspect here is the presence of wifi passwords (WPA pre-shared keys) in the wifi_new_config data as see in screenshot below. Yes, you can get wifi passwords from an adb backup now!

Figure 1 - Snippet of Wifi saved settings from com.android.providers.settings.data showing SSIDs & passwords
Here is a python script to read com.android.providers.settings.data and export the information to json files. Below you can see some of the data parsed by the this script for one of my test devices.


Figure 2 - Data from 'global' key
Figure 3 - Data from 'system' key


Figure 4 - Data from 'secure' key





Figure 5 - Data from 'softap_config' key
Figure 6 - Data from 'lock_settings' key

Tuesday, October 15, 2019

Part 2 - ADB keyvalue backups - Call Logs

This is Part 2 of the continuing blog series on ADB keyvalue backups. Today we focus on Call Log Backups. 

Call logs are backed up under 
<Backup.adb>/apps/com.android.calllogbackup/k/com.android.calllogbackup.data
They are backed up only if you specified the -keyvalue option and are available on non-rooted devices too.

This file follows the Key-Value Data format as outlined earlier in part 1. The Keys here are the call ids or serial number of calls, starting at 1 and sequentially rising. The Values are the individual call log records.

Here are the structures used in the Call Log record. All fields here are stored as Big Endian.

1. Text_Record

Position
Type
Description
00
ushort
field_length (in bytes)
02
char[field_length]
field data (text)

2. Call_Log

Position
Type
Description
00
uint
version, 0x03EF (1007) or 1005 seen
04
int64
timestamp
12
uint64
call duration in seconds
20
byte
is_phone_number_present
21
Text_Record
present if is_phone_number_present = 1
..
uint
call type
   1 = Incoming
   2 = Outgoing
   3 = Missed
   4 = voicemail
   5 = Rejected / Declined
   6 = Blocked
   7 = Answered_Externally
..
uint
number presentation
  1 = Allowed
  2 = Restricted
  3 = Unknown
  4 = Payphone
..
byte
is_servicename_present
..
Text_Record
present if is_ servicename_present = 1
..
byte
is_iccid_present
..
Text_Record
present if is_ iccid _present = 1
..
byte
is_own_num_present
..
Text_Record
present if is_ iccid _present = 1
..
byte[12]
unknown bytes, always 0
..
Text_Record
oem namespace string
..
byte[18]
unknown bytes
..
uint
block reason (only on version 1007)
   1 = Screening service
   2 = Direct to voicemail
   3 = Blocked number
   4 = Unknown number
   5 = Restricted number
   6 = Payphone
   7 = Not in contacts
..
byte[18]
unknown bytes (only on version 1007)

The screenshot below shows a raw record in the hex editor.

Figure 1 - Call log record showing some important fields

Using an 010 template to parse this information, it looks like this (below).

Figure 2 - Call log record data parsed in 010 editor
The level of detail on these records is great. There are call status codes known as Call Type (Missed, Incoming, Outgoing, ..) as well as a number Presentation code which is usually 1 (Allowed), although there are a few other values. Calls that show up on your phone as 'Private' numbers, will have presentation code 2 (Restricted). If you have enabled any call blocking features, then those show up too on blocked calls (known as block reason).

Code to automate this parsing

A python script has been created to parse call log records from the com.android.calllogbackup.data file, available here. The 010 template can be downloaded here.

Forensic Gems - Detecting Deleted call records

Since each call record has a key which is the call id or the serial number of the call, I performed an experiment to see if deleting intermittent call records would change this number. It turns out that the number does not change, in effect allowing us to detect deleted call records. This is visible in the screenshot below, where you can see call IDs (serial numbers) of 1 through 8 but its missing 4 and 7. Those are the ones I had manually deleted from the Call logs on the phone through the available feature in the Phone app. This was also tested on a real phone with several hundred call records going back several months and it appears to hold true there too.

Figure 3 - Output of callparser.py, made pretty in Excel showing missing call ids.
This can be useful knowing that there can be records which are missing, perhaps intentionally.

Stay tuned for Part 3, there is more good stuff in these key-value backups.

Monday, October 7, 2019

ADB keyvalue backups and the .data format

The ADB backup has been a very useful tool for getting data from Android phones, particularly those phones/devices that are otherwise not accessible due to lack of support by forensic software vendors or hardware/software issues with other methods.

There is however one feature which I do not see being used by any of the vendors or FOSS or any other guides out there. I am specifically talking about about ADB's backup feature to backup key-value pairs. According to one source, since Oreo (8.0), the keyvalue backups are now available via adb backup.

To get data with keyvalue pairs returned, you need to add the -keyvalue parameter to the adb backup command like. I like to use :

adb backup -all -shared -system -keyvalue -f file.adb

Keyvalue backups give some very good information otherwise not available in the adb backup.

So, where are the key-value backups located?


When viewing the adb tar archive, you will find one or more folders under each app's folder with names like k, sp, db, .. The k folder holds the key value backup, having file(s) which ends in the extension .data.
Figure 1 - Folders holding key-value .data files (this isn't all , there are many more)
The *.data files are located in the k folders usually having the same name as the package like com.android.calendar.data.

Parsing .data files

This consists of a series of records, each starting with 'Data', and having a key (name) and value (data). The format is as follows, all data in this structure is stored as little-endian:

Position
Type
Description
00
char[4]
‘Data’
04
uint
key_size
08
uint
data_size
12
char[key_size+1]
key_name
12 + key_size
char[]
pad to 4 byte boundary
..
char[data_size]
data
..
char[]
pad to 4 byte boundary


The value field can be different types depending on the data/database being backed up. It is different for different packages. You can find XML files, entire SQLITE databases in there, and also single byte true/false type settings.

In the screenshot below, you can see the key-value records as parsed out for com.android.vending.data. The 010 template for this is available here.

Figure 2 - Hex editor view of com.android.vending.data, showing 'Data' records parsed out using an 010 template

In the above example, the value types are mostly True/False. But most other databases have other custom structures embedded there, which need further parsing.

In part 2 of this ADB series, we explore the formats of call logs and other databases that are backed up.