Friday, July 20, 2018

APFS template for 010 Editor

For quite some time, I've been analyzing APFS mostly with custom python code, which is not very efficient and rather time consuming and is not visual. Since most people doing any kind of serious hex editing use the 010 Editor (as do I), this was long overdue.

I've created an 010 template, which is basically a port from the apfs.ksy project. This has taken quite a bit of time and I hope you find it useful. Not all structures are known, there are some parts that may be incorrect. This is a work in progress as more details about APFS emerge..

Link: https://github.com/ydkhatri/APFS_010/blob/master/apfs.010.bt

The template will not parse out the file system tree yet. With APFS this is challenging to do within 010's template capabilities as you cannot create local objects or classes and/or store temporary objects. The template does however define most of the structures and will follow most pointers (to other disk blocks and parse them) automatically when you start expanding the structures in the template viewer.

To use the template, simply load your APFS image (unencrypted only) into 010. Then edit the template to set the Apfs_Offset variable to the byte offset of wherever your APFS partition starts. Now run the template. The APFS start offset can be located easily by running the GPT template (which you can find on 010's website or in the program's template repository). The GPT template will give you the sector offset, multiply it by sector size (usually 512 or sometimes 4096) to get the byte offset (location) of the APFS partition.

Thursday, May 3, 2018

Bash sessions in macOS (and why you need to understand its working)

While all versions of macOS have provided bash_history for users, since macOS 10.11 (El Capitan), we get even more information on terminal history through the bash sessions files. This is not a replacement for the old .bash_history file which is still there.

There are several problems with bash_history - you cannot tell when any command in that file was run, the sequence of commands may not be right, and so on. For more on that, refer Hal Pomeranz's excellent talk - You don't know jack about Bash history

Even if there were no anomalies and only a single terminal was always in use, there is still the issue of how do I know which command was run when? With Bash sessions, macOS gives us more data to work with. Since El Capitan, every new terminal window will be tracked independently with a TERM_SESSION_ID which appears to be a randomly generated UUID.

Figure 1 - Fetching terminal's session id

Each session can also be restored when you shutdown and restart your machine with the "Reopen windows when logging back in" option set. Perhaps for this purpose, session history (a subset of bash history) is tracked and saved separately on a per session basis.

Figure 2 - Restored session

Show me the artifacts!

The location you want to go to is  /Users/<USER>/.bash_sessions

You will find 3 files for each session as seen in screenshot below.

Figure 3 - .bash_sessions folder contents


TERM_SESSION_ID.history    --> Contains session history
TERM_SESSION_ID.historynew --> Mostly blank/empty
TERM_SESSION_ID.session    --> Contains the last session resume date and time

Figure 4 - Sample .session file

Figure 5 - Sample .history file showing commands typed at terminal 


How this helps?

Some (but not all) of the problems associated with reading .bash_history are now gone.
Theoretically, as bash history is now also stored on a per session basis, this should make it trivial to track commands run in different windows (sessions). If you were expecting history for a single session in its .history file, then you thought wrong. The .history file contains all previous history (from earlier sessions) and then appended at the very end, the history for this session.

So can we reliably break apart commands per session? Is the sequence of commands intact? Let's run a small experiment to find out.

We create two sessions (2 terminal windows) and run a few commands in each session. Commands are interspersed, so we run a command in Session-1, then another in Session-2 and then again something in Session-1. We will try to see if order is maintained.

Session-1 started 9:44
Session-2 started 9:51
Figure 6 - Commands run with their sequence

Session-1 closed 9:57
Session-2 closed 9:59

Session-1 is closed first, followed by Session-2.  Here is a snippet of relevant metadata from the resulting files:

Figure 7 - Relevant metadata from stat command

Fun Facts

The start and stop time for a session is available if you look at the crtime (File Created time) for the .history and .historynew files. These are in bold in the screenshot above.

Created Time of TERM_SESSION_ID.historynew = Session created time
Created Time of TERM_SESSION_ID.history        = Session end time


Isolating session data

By comparing the data in various .history files (from different sessions), you can find out exactly which commands belong to a particular session. See pic below, where lines 1-181 (not shown) are from older history (other past sessions). Lines 182-184 are from Session-1 and are seen in its history file at the end. Session-2 (closed after Session-1) has the same format, ie, old session history with this session's history appended (lines 185-189).


Figure 8- .history files from Session-1 (Left) and Session-2 (Right)

This is easily done in code and the mac_apt BASHSESSIONS plugin parses this information to break out the individual commands per session, along with session start and stop time.

While you still cannot get the exact time when an individual command was run, the sessions functionality does give you a very good narrowed time frame to work with. While we do not have the absolute order of commands ("cp -h" was run before "printenv"), we do have a narrowed time-frame for the set of commands ("cp-h" run between 9:51-9:59 and "printenv" run between 9:44-9:57). This is a big thing for analysts and investigators!


Friday, February 2, 2018

Reading Notes database on macOS

The notes app comes built-in with every OSX/macOS release since OSX 10.8 (Mountain Lion). It is great for quick notes and keeps notes synced in the cloud. It can have potentially useful tidbits of user information in an investigation.

Artifact breakdown  & Forensics

Locations 

Depending on version of macOS, this can vary. Sometimes there is more than one database, probably as a result of an upgrade! But only one is actively used at any given time (not both). So far, we haven't seen any duplicate data when two are present.

Location 1

/Users/<USER>/Library/Containers/com.apple.Notes/Data/Library/Notes/
In here, databases will be named as one of the following:
NotesV1.storedata ← Mountain Lion  (Thanks Geoff Black)
NotesV2.storedata ← Mavericks
NotesV4.storedata ← Yosemite
NotesV6.storedata ← Elcapitan & Sierra
NotesV7.storedata ← HighSierra
If a note has an attachment, then the attachment is usually stored at the following location:
/Users/<USER>/Library/Containers/com.apple.Notes/Data/Library/CoreData/Attachments/UUID/

There does not appear to be much difference between the database types. The following tables have been seen.
     
Tables for NotesV2
Tables for NotesV6

Each note can be associated with either a local account or an online one. The following account information can be obtained.  

Account email address, id and username from ZACCOUNT table

Individual note data is stored in ZNOTEBODY and the rest of the tables provide information about note parent folder, sync information, and attachment locations. 

Notes converts all data to HTML as seen below.

ZNOTES Table with note Html content
The graphic below shows how you can find and resolve note attachments to their locations on disk. If an attachment is present, ZNOTEBODY.ZHTMLSTRING will contain the UUID of that which can be matched up to the ZATTACHMENT.ZCONTENTID to get a binary plist blob. When parsed, you can find the full path to the attachment in the plist.

Resolving attachment location

The dates and times fetched are Mac Absolute time, which is the number of seconds since 1/1/2001.

Reading the data

The following SQL query will pull out most pertinent information from this database:

SELECT n.Z_PK as note_id, datetime(n.ZDATECREATED + 978307200, 'unixepoch') as created, datetime(n.ZDATEEDITED + 978307200, 'unixepoch') as edited, n.ZTITLE, (SELECT ZNAME from ZFOLDER where n.ZFOLDER=ZFOLDER.Z_PK) as Folder,(SELECT zf2.ZACCOUNT from ZFOLDER as zf1  LEFT JOIN ZFOLDER as zf2 on (zf1.ZPARENT=zf2.Z_PK) where n.ZFOLDER=zf1.Z_PK) as folder_parent,ac.ZEMAILADDRESS as email, ac.ZACCOUNTDESCRIPTION, b.ZHTMLSTRING, att.ZCONTENTID, att.ZFILEURLFROM ZNOTE as nLEFT JOIN ZNOTEBODY as b ON b.ZNOTE = n.Z_PKLEFT JOIN ZATTACHMENT as att ON att.ZNOTE = n.Z_PKLEFT JOIN ZACCOUNT as ac ON ac.Z_PK = folder_parent

Location 2

/Users/<USER>/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite
This one has been seen on El Capitan, Sierra and HighSierra. Attachments are stored in the Media folder located here:
/Users/<USER>/Library/Group Containers/group.com.apple.notes/Media/<UUID>/
Here UUID is the unique identifier for each attachment. The database scheme is different here.

NoteStore.sqlite in HighSierra

NotesStore.sqlite in ElCapitan





Only account name and identifier (UUID) is available. To get full account information, this will need to be correlated with the account info database stored elsewhere.
ZICLOUDSYNCINGOBJECT account info
Note data is available in the ZICNOTEDATA table. 

ZICNOTEDATA table
That ZICNOTEDATA.ZDATA blob is gzip compressed. Upon decompression, it reveals the note data stored in a proprietary unknown binary format. As seen below, you can spot the text, its formatting information and attachment info.

ZDATA gzipped blob showing gzip signature (1F8B08)
Uncompressed ZDATA blob showing text, formatting and attachment info

Most notable in this database is the presence of several timestamps-
Note Title Modified → ZDATEFORLASTTITLEMODIFICATION
Note Created → ZCREATIONDATE
Note Modified → ZMODIFICATIONDATE1
Attachment Modified → ZMODIFICATIONDATE
Attachment Preview Updated → ZPREVIEWUPDATEDATE

Reading NoteStore.sqlite


The following SQL query will pull out most pertinent information from this database:
SELECT n.Z_12FOLDERS as folder_id , n.Z_9NOTES as note_id, d.ZDATA as data,
c2.ZTITLE2 as folder,
datetime(c2.ZDATEFORLASTTITLEMODIFICATION + 978307200, 'unixepoch') as folder_title_modified,
datetime(c1.ZCREATIONDATE + 978307200, 'unixepoch') as created,
datetime(c1.ZMODIFICATIONDATE1 + 978307200, 'unixepoch')  as modified,
c1.ZSNIPPET as snippet, c1.ZTITLE1 as title, c1.ZACCOUNT2 as acc_id,
c5.ZACCOUNTTYPE as acc_type, c5.ZIDENTIFIER as acc_identifier, c5.ZNAME as acc_name,
c3.ZMEDIA as media_id, c3.ZFILESIZE as att_filesize,
datetime(c3.ZMODIFICATIONDATE + 978307200, 'unixepoch') as att_modified,
datetime(c3.ZPREVIEWUPDATEDATE + 978307200, 'unixepoch') as att_previewed,
c3.ZTITLE as att_title, c3.ZTYPEUTI, c3.ZIDENTIFIER as att_uuid,
c4.ZFILENAME, c4.ZIDENTIFIER as media_uuid
FROM Z_12NOTES as n
LEFT JOIN ZICNOTEDATA as d ON d.ZNOTE = n.Z_9NOTES
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c1 ON c1.Z_PK = n.Z_9NOTES
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c2 ON c2.Z_PK = n.Z_12FOLDERS
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c3 ON c3.ZNOTE = n.Z_9NOTES
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c4 ON c3.ZMEDIA = c4.Z_PK
LEFT JOIN ZICCLOUDSYNCINGOBJECT as c5 ON c5.Z_PK = c1.ZACCOUNT2
ORDER BY note_id
 
On HighSierra, use this query:
SELECT n.Z_PK, n.ZNOTE as note_id, n.ZDATA as data,c3.ZFILESIZE,c4.ZFILENAME, c4.ZIDENTIFIER as att_uuid,c1.ZTITLE1 as title, c1.ZSNIPPET as snippet, c1.ZIDENTIFIER as noteID,datetime(c1.ZCREATIONDATE1, 'unixepoch') as created, datetime(c1.ZLASTVIEWEDMODIFICATIONDATE, 'unixepoch'), datetime(c1.ZMODIFICATIONDATE1, 'unixepoch') as modified,c2.ZACCOUNT3, c2.ZTITLE2 as folderName, c2.ZIDENTIFIER as folderID,c5.ZNAME as acc_name, c5.ZIDENTIFIER as acc_identifier, c5.ZACCOUNTTYPEFROM ZICNOTEDATA as nLEFT JOIN ZICCLOUDSYNCINGOBJECT as c1 ON c1.ZNOTEDATA = n.Z_PKLEFT JOIN ZICCLOUDSYNCINGOBJECT as c2 ON c2.Z_PK = c1.ZFOLDERLEFT JOIN ZICCLOUDSYNCINGOBJECT as c3 ON c3.ZNOTE= n.ZNOTELEFT JOIN ZICCLOUDSYNCINGOBJECT as c4 ON c4.ZATTACHMENT1= c3.Z_PKLEFT JOIN ZICCLOUDSYNCINGOBJECT as c5 ON c5.Z_PK = c1.ZACCOUNT2ORDER BY note_id
If you are looking for an automated way to read this, use mac_apt, the NOTES plugin will parse it.