tag:blogger.com,1999:blog-12646112603227784862024-03-12T19:00:14.366-04:00Yogesh Khatri's forensic blogAll things forensic and security relatedYogesh Khatrihttp://www.blogger.com/profile/03726664886311447808noreply@blogger.comBlogger58125tag:blogger.com,1999:blog-1264611260322778486.post-5679719392134092972022-11-04T10:35:00.001-04:002022-11-04T11:18:27.300-04:00Reading OneDrive Logs Part 2<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggE37njXwrHilRu30QZefGSCb4IFl9yvpEkYJgLLuL_EqXniSGyunxngqRaeQLzhzKtJ_0kmcZRelDVv40Z58JwXatHDsrqvRuhd7WFVAxhII30QGAa3s-txuVs85Krv6fKE5BJ_wtIDiQciXVDL1zCwdkFeeJEO7PdSYhTAiS6BNonYilLGrDxdD5/s658/OneDrive.jpeg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="370" data-original-width="658" height="179" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggE37njXwrHilRu30QZefGSCb4IFl9yvpEkYJgLLuL_EqXniSGyunxngqRaeQLzhzKtJ_0kmcZRelDVv40Z58JwXatHDsrqvRuhd7WFVAxhII30QGAa3s-txuVs85Krv6fKE5BJ_wtIDiQciXVDL1zCwdkFeeJEO7PdSYhTAiS6BNonYilLGrDxdD5/w320-h179/OneDrive.jpeg" width="320" /></a></div>In the last OneDrive <a href="https://www.swiftforensics.com/2022/02/reading-onedrive-logs.html" target="_blank">blog post</a>, I outlined how the ODL file format is structured. A working version of an ODL parser was also created to read these files. One key detail was how personal file/folder, location or credential identifying strings were obfuscated with the original values stored in the <i>ObfuscationStringMap.txt </i>file. <br /><p></p><p>However some time in April 2022, Microsoft decided to change the way the obfuscation worked and the parser no longer worked (the unobfuscation part). </p><h3 style="text-align: left;">What changed?</h3><p>OneDrive now appeared to encrypt the data and the <i>ObfuscationStringMap.txt</i> is no longer used. The file may still exist on older installations, but newer ones include a different file.</p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXnShQstnol7vlvlBmVWkANjcVPKIBUst4EzaQgJ2e-Mqkf_pSsQbcFcTYBksl6EH2qaoOe9Zba3nqaZQkZSITL8vK0vM0YJWIua7c0-wZ7FPuEnxA-5-lI008TAH6bJWPUP4rT360VQKI5S8JCheYSO4HwWObCnqvhB0uKkP3_DgHiXGuPZMVmn37/s539/business1.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="297" data-original-width="539" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXnShQstnol7vlvlBmVWkANjcVPKIBUst4EzaQgJ2e-Mqkf_pSsQbcFcTYBksl6EH2qaoOe9Zba3nqaZQkZSITL8vK0vM0YJWIua7c0-wZ7FPuEnxA-5-lI008TAH6bJWPUP4rT360VQKI5S8JCheYSO4HwWObCnqvhB0uKkP3_DgHiXGuPZMVmn37/w400-h220/business1.PNG" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Contents of \AppData\Local\Microsoft\OneDrive\logs\Business1 </i><i>folder</i></td></tr></tbody></table><br /><p></p><p>As seen in <i>Figure 1</i> above, there is a new file called <b>general.keystore</b>. This file's format is JSON that can be easily read and apparently holds the key to decrypt the encrypted content as a base64 encoded string. </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisx5VbM0v1Zqhorx4vuvv6ZuLNa9zKaCpTTcjheqKa8ZuIh8LYrTPpFegGfpQs5IKCEZoAyLmEUVC2-cdaFISkL1PCZexAw9_u-TgTdcVn9txjzQeWTvMo5J_QHh8l06GJSZ43fCEm4e24Gb_JFpQG_4f3D5echypwxc4HNwVgStn9J29ht383nstI/s764/general.keystore%20contents.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="158" data-original-width="764" height="83" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisx5VbM0v1Zqhorx4vuvv6ZuLNa9zKaCpTTcjheqKa8ZuIh8LYrTPpFegGfpQs5IKCEZoAyLmEUVC2-cdaFISkL1PCZexAw9_u-TgTdcVn9txjzQeWTvMo5J_QHh8l06GJSZ43fCEm4e24Gb_JFpQG_4f3D5echypwxc4HNwVgStn9J29ht383nstI/w400-h83/general.keystore%20contents.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Sample general.keystore contents</i></td></tr></tbody></table><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;">Time for some Reverse Engineering</h3><p>With a little bit of digging around with IDA Pro on the<i> LoggingPlatform.dll </i>file from OneDrive, we can see the <a href="https://learn.microsoft.com/en-us/windows/win32/api/bcrypt/" target="_blank">BCrypt Windows APIs</a> being used in this file. <i>Note, this is not the bcrypt hash algorithm which bears the same name!</i></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-EmEUEaCIa6bjI3lXHo5YltbbrQRkthhIv-Usog3XBPjk29K8JqOq7JIWiVEfGgB-DKPtUvrHsJi_rp83RG9Cx_EtvLgLeM5BGW4tJEm3mLcYxcFHDKvcTipj8C_XMNf5AN7AlGY4U9LAiZBxcww2SL-m61i4dRdQNIlup0fxyndQQ6ujLDubENBv/s1096/imports.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="310" data-original-width="1096" height="182" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-EmEUEaCIa6bjI3lXHo5YltbbrQRkthhIv-Usog3XBPjk29K8JqOq7JIWiVEfGgB-DKPtUvrHsJi_rp83RG9Cx_EtvLgLeM5BGW4tJEm3mLcYxcFHDKvcTipj8C_XMNf5AN7AlGY4U9LAiZBxcww2SL-m61i4dRdQNIlup0fxyndQQ6ujLDubENBv/w640-h182/imports.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - BCrypt* Imports in LoggingPlatform.dll</i></td></tr></tbody></table><br /><p></p><p>Jumping to where these functions are used, it is quickly apparent that the encryption used is <b>AES</b> in <b>CBC</b> (Cipher Block Chaining) mode with a key size of 128 bits. </p><div class="separator" style="clear: both; text-align: center;"><br /></div><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJg1JZXTj_9xkfWhrZK5x5zdHpBTS8CIDGJldFTADuH1ke264OEA7kCizM_2OiQvo97Unf2C8TuLXTeoZeim78d9eHahaVx40q5OwqMJ8ZwnCy-I9znXdXhuGAE4WNm7oGJqqA7zki0UKTmI-dzb_bF7Opw_puu5JAdiN37jFU1lzDIAWuHw6SNy01/s1190/IDA_assembly.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="734" data-original-width="1190" height="394" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJg1JZXTj_9xkfWhrZK5x5zdHpBTS8CIDGJldFTADuH1ke264OEA7kCizM_2OiQvo97Unf2C8TuLXTeoZeim78d9eHahaVx40q5OwqMJ8ZwnCy-I9znXdXhuGAE4WNm7oGJqqA7zki0UKTmI-dzb_bF7Opw_puu5JAdiN37jFU1lzDIAWuHw6SNy01/w640-h394/IDA_assembly.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 4 - IDA Pro Disassembly</i></td></tr></tbody></table><br />In the above snippet, we can see the call to <span style="font-family: courier;">BCryptAlgorithmProvider</span> and then if successful, a call to <span style="font-family: courier;">BCryptSetProperty</span> function which has the following syntax:<div><br /><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><div><span style="color: #666666; font-family: courier;">NTSTATUS </span><span style="color: #3d85c6; font-family: courier;">BCryptSetProperty</span><span style="color: #666666; font-family: courier;">(</span></div></div><div><div><span style="color: #666666; font-family: courier;"> [in, out] BCRYPT_HANDLE hObject,</span></div></div><div><div><span style="color: #666666; font-family: courier;"> [in] LPCWSTR pszProperty,</span></div></div><div><div><span style="color: #666666; font-family: courier;"> [in] PUCHAR pbInput,</span></div></div><div><div><span style="color: #666666; font-family: courier;"> [in] ULONG cbInput,</span></div></div><div><div><span style="color: #666666; font-family: courier;"> [in] ULONG dwFlags</span></div></div><div><div><span style="color: #666666; font-family: courier;">);</span></div></div></blockquote><div><div><br /></div><p>Without delving into too many boring assembly details, I'll skip to the relevant parts...</p><p>For each string to be encrypted, OneDrive initialises a new encryption object with the key that is stored in the <i>general.keystore </i>file, then encrypts the string and disposes of the encryption object. The encrypted blob is then base64 encoded and written out to the log the obfuscated string. There are a few other quirks along the way, such as replacement of the characters <b><span style="color: #2b00fe; font-family: courier;">/ </span></b>and <span style="color: #2b00fe; font-family: courier;"><b style="background-color: white;">+ </b></span>with <b><span style="background-color: white; color: #6aa84f; font-family: courier;">_</span></b> and <b><span style="color: #6aa84f; font-family: courier;">- </span></b>respectively, as the former can appear in base64 text but are also used in URLs to make it parseable later.</p><p><br /></p><h3 style="text-align: left;"><b>Why the change?</b></h3><p>In the previous iteration of ODL (when the ObfuscationStringMap was used), there were instances where the same key (3 word combination) was often repeated in the file making it difficult or impossible to know which value to use as its replacement to get the original string.</p><p>Using encryption in place and not using a lookup table does appear to be a more robust scheme which eliminates the above issue. It does use some more disk space as the encrypted blob will always be a multiple of 16 bytes (128 bits) as this is block based encryption. In other words, it's inefficient for small text (less than 10 bytes). </p><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;">Updated code</h3></div></div>The python ODL parser has been updated to accomodate this new format, and works with both the old and new versions. It is available <a href="https://github.com/ydkhatri/OneDrive">here</a>.Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-25263230082723025812022-02-13T07:34:00.001-05:002022-02-13T07:34:51.160-05:00Reading OneDrive Logs<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgbz575Gx4lsEon88e8zCOI7Sp4wGTpe6QfYKr-ASXoTysVrcuYFZKCiWax6BlIL0IivmbOKq2G1sS3u3tYMi55PPQ-MAqrU7gnMooOg79YlMKSUfis2GiZG5q7i-pnaWhsTpVRE_9jhvzwozwF2tLTo0TTWRD4sGQtL3euL-PQVD13b_bVOKwHZwyQ=s1983" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="625" data-original-width="1983" height="101" src="https://blogger.googleusercontent.com/img/a/AVvXsEgbz575Gx4lsEon88e8zCOI7Sp4wGTpe6QfYKr-ASXoTysVrcuYFZKCiWax6BlIL0IivmbOKq2G1sS3u3tYMi55PPQ-MAqrU7gnMooOg79YlMKSUfis2GiZG5q7i-pnaWhsTpVRE_9jhvzwozwF2tLTo0TTWRD4sGQtL3euL-PQVD13b_bVOKwHZwyQ=s320" width="320" /></a></div>Due to the popularity of OneDrive, it has become an important source of evidence in forensics. Last week, Brian Maloney posted about his <a href="https://www.sans.org/blog/recreating-onedrive-s-folder-structure-from-usercid-dat/" target="_blank">research on reconstructing the folder tree from the <i>usercid</i>.<i>dat</i> files</a>, and also <a href="https://github.com/Beercow/OneDriveExplorer" target="_blank">provided a script</a> to do so. In this brief post, we explore the format of OneDrive Logs, and provide a tool to parse them. In subsequent posts, I will showcase use case scenarios. <p></p><h3 style="text-align: left;">Where to find them?</h3><p>OneDrive logs have the extension <b>.odl</b> and are found in the user's profile folder at the following locations:</p><p><i>On Windows - </i></p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><b style="background-color: #fcff01;">C:\Users\<USER>\AppData\Local\Microsoft\OneDrive\logs\</b></p></blockquote><p><i>On macOS -<b> </b></i></p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><b style="background-color: #fcff01;">/Users/<USER>/Library/Logs/OneDrive/</b></p></blockquote><p>At these locations, there are usually 3 folders - <i>Common, Business1 </i>and<i> Personal</i>, each containing logs. As the name suggests <i>Business1</i> is the <i>OneDrive for Business</i> version. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgPS-jJFCPgnnoRq-AiZjU5xBvCXZHic-YQdvqHf4WSIorQkfKHD6WVMlX_Jc6RWY3MIR8WqzHIn0VZ-jUEKF07UwWahAEakc4WkucSauzI1UG8BI74QQkueIR3byqfngnCAucm699cKJb42TCy7W0j_UI0smDjgFZSCeHBeLUMDknbEbbA_HiooMvs=s453" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="453" data-original-width="293" src="https://blogger.googleusercontent.com/img/a/AVvXsEgPS-jJFCPgnnoRq-AiZjU5xBvCXZHic-YQdvqHf4WSIorQkfKHD6WVMlX_Jc6RWY3MIR8WqzHIn0VZ-jUEKF07UwWahAEakc4WkucSauzI1UG8BI74QQkueIR3byqfngnCAucm699cKJb42TCy7W0j_UI0smDjgFZSCeHBeLUMDknbEbbA_HiooMvs=s16000" /></a></div><div class="separator" style="clear: both; text-align: center;"><i>Figure 1 - Contents of Business1 folder</i></div><br /><p>The .odl file is the currently active log, while the .odlgz files are older logs that have been compressed. Depending on which folder (and OS) you are in, you may also see .odlsent and .aodl files, which have a similar format.</p><h3 style="text-align: left;">What is in there?</h3><p>These are binary files, and cannot be directly viewed in a text editor. Here is what a .odl file looks like in a hex editor.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjMjUAfGlD1-JDMNw66ONw2q0xlDw4WQKnFVkxen2TL7n_3m_WZtxLTx_BBj3sZyWSyV86RDIciHqIszD-AUP2dC4LXPvebA92gbMXdf6WhYlTSVAs9IaBh1WG1r9n3laLlr6zHEAUpetyq4TFl0af9RPR5KO5_KmgNYpFtTO1-yG6Q34mGl5-Z5vmJ=s605" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="503" data-original-width="605" height="333" src="https://blogger.googleusercontent.com/img/a/AVvXsEjMjUAfGlD1-JDMNw66ONw2q0xlDw4WQKnFVkxen2TL7n_3m_WZtxLTx_BBj3sZyWSyV86RDIciHqIszD-AUP2dC4LXPvebA92gbMXdf6WhYlTSVAs9IaBh1WG1r9n3laLlr6zHEAUpetyq4TFl0af9RPR5KO5_KmgNYpFtTO1-yG6Q34mGl5-Z5vmJ=w400-h333" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><i>Figure 2 - .odl file in a hex editor</i></div><p>It is a typical binary file format with a 256 byte header and data blocks that follow. Upon first inspection, it seems to be a log of all important function calls made by the program. Having this low level run log can be useful in certain scenarios where you don't have other logging and need to prove upload/download or synchronisation of files/folders or even a discovery of items which no longer exist on disk/cloud. </p><p>You do notice some funny looking strings (highlighted in <span style="color: red;">red</span>). More on that later.. </p><h3 style="text-align: left;">The Format</h3><p>With a bit of reverse engineering, the header format is worked out as follows:</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div style="text-align: left;"><span style="font-family: courier;"><span style="color: #2b00fe;">struct</span> {</span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">char</span> signature[8]; <span style="color: #93c47d;">// EBFGONED</span></span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">uint32</span> unk_version; <span style="color: #93c47d;">// value seen = 2</span></span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">uint32</span> unknown2;</span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">uint64</span> unknown3; <span style="color: #93c47d;">// </span></span><span style="color: #93c47d; font-family: courier;">value seen =</span><span style="color: #93c47d; font-family: courier;"> </span><span style="color: #93c47d; font-family: courier;">0</span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">uint32</span> unknown4; <span style="color: #93c47d;">// </span></span><span style="color: #93c47d; font-family: courier;">value seen =</span><span style="color: #93c47d; font-family: courier;"> </span><span style="color: #93c47d; font-family: courier;">1</span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">char</span> one_drive_version[0x40];</span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">char</span> os_version[0x40];</span></div><div style="text-align: left;"><span style="font-family: courier;"> <span style="color: #2b00fe;">byte</span> reserved[0x64];</span></div><div style="text-align: left;"><span style="font-family: courier;">} Odl_header;</span></div></blockquote><div style="text-align: left;"><span style="font-family: courier;"><br /></span></div>The structures for the data blocks are as follows:<div><br /></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><div><div><span style="font-family: courier;"><span style="color: #2b00fe;">struct</span> {</span></div></div></div><div><div><div><span style="font-family: courier;"> <span style="color: #2b00fe;">uint64</span> signature; <span style="color: #93c47d;">// CCDDEEFF 0000000</span></span></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="color: #2b00fe; font-family: courier;">uint64</span><span style="font-family: courier;"> timestamp; <span style="color: #93c47d;">// Unix Millisecond time</span></span></div></div></div><div><div><div><span style="font-family: courier;"> <span style="background-color: white;"><span style="color: #2b00fe;">uint32</span></span> unk1;</span></div></div></div><div><div><div><span style="font-family: courier;"> <span style="color: #2b00fe;">uint32</span> unk2;</span></div></div></div><div><div><div><span style="font-family: courier;"> <span style="color: #2b00fe;">byte </span> unk3_guid[16];</span></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="background-color: white; color: #2b00fe; font-family: courier;">uint32</span><span style="font-family: courier;"> unk4;</span></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="background-color: white; color: #2b00fe; font-family: courier;">uint32</span><span style="font-family: courier;"> unk5; <span style="color: #93c47d;">// mostly 1</span></span></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="background-color: white; color: #2b00fe; font-family: courier;">uint32</span><span style="font-family: courier;"> data_len;</span></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="background-color: white; color: #2b00fe; font-family: courier;">uint32</span><span style="font-family: courier;"> unk6; <span style="color: #93c47d;">// mostly 0</span></span></div></div></div><div><div><div><span style="font-family: courier;"> <span style="color: #2b00fe;">byte</span> data[data_len];</span></div></div></div><div><div><div><span style="font-family: courier;">} Data_block;</span></div></div></div><div><div><div><br /></div></div></div><div><div><span style="color: #2b00fe; font-family: courier;">struct</span><span style="font-family: courier;"> {</span></div></div><div><div><div><span style="font-family: courier;"> </span><span style="background-color: white; color: #2b00fe; font-family: courier;">uint32</span><span style="font-family: courier;"> </span><span style="font-family: courier;">code_file_name_</span><span style="font-family: courier;">len;</span></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="color: #2b00fe; font-family: courier;">char</span><span style="font-family: courier;"> </span><span style="font-family: courier;">code_file_name[</span><span style="font-family: courier;">code_file_name_</span><span style="font-family: courier;">len]</span><span style="font-family: courier;">;</span></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="background-color: white; color: #2b00fe; font-family: courier;">uint32</span><span style="font-family: courier;"> unknown;</span></div></div></div><div><div><div><div><span style="font-family: courier;"> </span><span style="background-color: white; color: #2b00fe; font-family: courier;">uint32</span><span style="font-family: courier;"> </span><span style="font-family: courier;">code_function_name</span><span style="font-family: courier;">_</span><span style="font-family: courier;">len;</span></div></div></div></div><div><div><div><div><span style="font-family: courier;"> </span><span style="color: #2b00fe; font-family: courier;">char</span><span style="font-family: courier;"> </span><span style="font-family: courier;">code_function_name</span><span style="font-family: courier;">[</span><span style="font-family: courier;">code_function_name</span><span style="font-family: courier;">_</span><span style="font-family: courier;">len]</span><span style="font-family: courier;">;</span></div></div></div></div><div><div><div><span style="font-family: courier;"> </span><span style="color: #2b00fe; font-family: courier;">byte</span><span style="font-family: courier;"> parameters[];</span></div></div></div><div><div><span style="font-family: courier;">} Data;</span></div></div></blockquote><div><div><div></div></div></div><div><div><br /></div><div><i>In case of .odlgz files, the Odl_header is the same, followed by a single gzip compressed blob. The blob can be uncompressed to parse the Data_block structures.</i></div><div><br /></div><div>Now, we can try to interpret the data. Leaving aside the few unknowns, the data block mainly consists of a timestamp (when event occurred), the function name that was called, the code file that function resides in and the parameters passed to the function. The parameters can be of various types like int, char, float, etc.. and that part hasn't been fully reverse engineered yet, but simply extracting the strings gives us a lot of good information. However the strings are obfuscated!</div><div><br /></div><h3 style="text-align: left;">Un-obfuscating the strings</h3><div>Since Microsoft uploads these logs to their servers for telemetry and debugging, they obfuscate anything that is part of a file/folder name or url string or username. However file extensions are not obfuscated. The ways this works is that the data identified to obfuscate is replaced by a word which is stored in a dictionary. The dictionary is available as the file <i>ObfuscationStringMap.txt, </i>usually in the same folder as the .odl files. To un-obfuscate, one simple needs to find and replace the strings with their original versions. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhpZEE7B-IP4brHzoiuKSVQPBoDejPdilsuElUXIP4gx2jzntvOZ2X2EKzUsn8LjOiSpBuYQ5g9o_eL6lFNdIhIcwQiknN_x36tbWtOHVch1hIEjLr3-RNl3pu2kaMrUQ0nswyUTzJ11vQxychzggx-zpNlwq9n6jahU5rhVGzQVvkWaeqaE58KcOl0=s701" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="384" data-original-width="701" height="350" src="https://blogger.googleusercontent.com/img/a/AVvXsEhpZEE7B-IP4brHzoiuKSVQPBoDejPdilsuElUXIP4gx2jzntvOZ2X2EKzUsn8LjOiSpBuYQ5g9o_eL6lFNdIhIcwQiknN_x36tbWtOHVch1hIEjLr3-RNl3pu2kaMrUQ0nswyUTzJ11vQxychzggx-zpNlwq9n6jahU5rhVGzQVvkWaeqaE58KcOl0=w640-h350" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><i>Figure 3 - Snipped of ObfuscationStringMap.txt</i></div><br /><div>This is a tab separated file, stored as either UTF-8 or UTF-16LE depending on whether you are running macOS or Windows. </div><div><br /></div><div>Now, referring back to the original funny looking string in Figure 2 - </div><div>/<span style="background-color: #e06666;">LeftZooWry</span>/<span style="background-color: #e06666;">LogOneMug</span>/<span style="background-color: #e06666;">HamUghVine</span>/<span style="background-color: #e06666;">MuchDownRich</span>/<span style="background-color: #e06666;">QuillRodEgg</span>/<span style="background-color: #e06666;">KoiWolfTad</span>/<span style="background-color: #e06666;">LawFlyOwl</span>.txt</div><div> .. after un-obfuscating becomes ..</div><div>/<span style="background-color: #04ff00;">Users</span>/<span style="background-color: #04ff00;">ykhatri</span>/<span style="background-color: #04ff00;">Library</span>/<span style="background-color: #04ff00;">Logs</span>/<span style="background-color: #04ff00;">OneDrive</span>/<span style="background-color: #04ff00;">Business1</span>/<span style="background-color: #04ff00;">telemetry-dll-ramp-value</span>.txt</div><div><br /></div><div>It is important to note that since extensions are not obfuscated, they can still provide valuable intel even if some or all of the parts in the path cannot be decoded.</div><div><br /></div><div>Now this process seems easy, however not all obfuscated strings are file/folder names or parts of paths/urls. Some are multi line strings. Another problem is that the words (or keys to the dictionary) are reused! So you might see the same key several times in the <i>ObfuscationStringMap. </i>The thing to remember is that new entries get added at the top in this file, not at the bottom, so when reading a file, the first occurrence of a key should be the latest one. <i> </i>Also sometimes, the key is not found, as it's cleaned out after a period of time. Also there is no way to tell if an entry in the dictionary is stale or valid for a specific log file being parsed. All of this just means that the decoded strings <i>need to be taken with a grain of salt.</i> </div><div><br /></div><div>Based on the above, a python script to parse the ODL logs is available <a href="https://github.com/ydkhatri/OneDrive" target="_blank">here</a>. A snippet of the output produced by the script is shown below.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjaY91_r9Mj4DaKg-E4UsQ_kiXtHHTOfPR71vlho1N5WFU4RRPDdeq3LgaRO1XlQDZA550z2uFU9_iAK95kkYZZx0p23roG1EbWESt9XGMBmKfMzsxB8KCMrCH3da9d5qP6zZIsc8QmHh0TBZvOL_1-5DxCHNLKotGIJ9PQiioi2IucqjeuRLgvByWc=s2550" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1004" data-original-width="2550" height="252" src="https://blogger.googleusercontent.com/img/a/AVvXsEjaY91_r9Mj4DaKg-E4UsQ_kiXtHHTOfPR71vlho1N5WFU4RRPDdeq3LgaRO1XlQDZA550z2uFU9_iAK95kkYZZx0p23roG1EbWESt9XGMBmKfMzsxB8KCMrCH3da9d5qP6zZIsc8QmHh0TBZvOL_1-5DxCHNLKotGIJ9PQiioi2IucqjeuRLgvByWc=w640-h252" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><i>Figure 4 - output snippet</i></div><br /><div><br /></div><div>In a subsequent post, we'll go through the items of interest in these logs like Account linking/unlinking, uploads, downloads, file info, etc.. </div></div>Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com1tag:blogger.com,1999:blog-1264611260322778486.post-24309036209974586112021-01-09T15:26:00.003-05:002021-01-09T15:27:46.316-05:00Gboard has some interesting data..<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGH6UiuvB5AUzqwn5Tbn8_QiB1MHGtqbfprSYbwmOzpOY4Jqf0010F6QArD2rISs-rdCXIriOnrjCmYurBEoMRmzp-kyfAmLFOgWpOKjWltiUDuXKVbvRJOYg1BNIrWhcmt4_TFWioQUM/s180/gboard.webp" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="180" data-original-width="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGH6UiuvB5AUzqwn5Tbn8_QiB1MHGtqbfprSYbwmOzpOY4Jqf0010F6QArD2rISs-rdCXIriOnrjCmYurBEoMRmzp-kyfAmLFOgWpOKjWltiUDuXKVbvRJOYg1BNIrWhcmt4_TFWioQUM/s0/gboard.webp" /></a></div>Gboard - the Google Keyboard, is the default keyboard on Pixel devices, and overall has been installed over a billion times according to the <a href="https://play.google.com/store/apps/details?id=com.google.android.inputmethod.latin&hl=en_US&gl=US" target="_blank">Play Store</a>.<br /><p></p><p>Although not the default on most non-Google brands, it is a popular app installed by foreign language users because of its good support and convenience of use particularly with dozens of Asian and Indian languages.</p><p>As a keyboard app, it monitors and analyzes your keystrokes, offering suggestions and corrections for spelling and grammar, sentence completion and even emoji suggestions. </p><p><b>Now for the interesting part.</b> Since the last few versions, it also retains a lot of data (ie, user keystrokes!) in its cache. This is at least seen from the version from Jan 2020 (v 8.3.x). From a DFIR perspective, that is GOLD. For a forensic examiner, this can possibly show you data that was typed by the user on an app that is now deleted, or show messages typed that were then deleted, or messages from apps that have the disappearing message feature turned on! Or data entered into fields on web pages/online apps (that wouldn't be stored locally at all). Also for some apps that don't track when a particular item was created/modified, this could be useful.</p><p><i><u>Note</u> - The Signal app wasn't specifically tested to see if data from that app is retained, but based on what we can see here, it seems likely those messages would end up here too. All testing was on a Pixel 3 running latest Android 11 using the default keyboard, and default settings. This was also verified on other earlier taken images. <a href="https://thebinaryhick.blog/2020/02/15/android-10-image-now-available/" target="_blank">Josh Hickman's Android 10 Pixel 3 image</a> was also used, and Josh was able to verify that Telegram and </i><i>WhatsApp</i><i> sent messages were present here. The specific versions of Gboard databases studied were:</i></p><p></p><ul><li><i>8.3.6.250752527 (on Android 10)</i></li><li><i>8.8.10.277552084 (on Android 10)</i></li><li><i>10.0.02.338070508 (on Android 11)</i></li></ul><p></p><h3 style="text-align: left;">Location</h3>Gboard's app data (sandbox) folder is located here:<br /><br /><span style="background-color: #fcff01;">/data/data/com.google.android.inputmethod.latin/databases/</span><br /><br />Here you might see a number of databases that start with trainingcache*. These are the files that contain the caches. <p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYH1mcFz192uPEZEUpQPd3RZ-w_iSW2fI4HS8b_dxserDlWEvtg7ySt7SPZER_HyfuHeOvi6wawFghhzWcpkgizXwGPmX7Hkb51s2aHGoPfPg4YIXMkytciNmmUo3NaNdC7t6IkRMlTZA/s232/gboard_data.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="194" data-original-width="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYH1mcFz192uPEZEUpQPd3RZ-w_iSW2fI4HS8b_dxserDlWEvtg7ySt7SPZER_HyfuHeOvi6wawFghhzWcpkgizXwGPmX7Hkb51s2aHGoPfPg4YIXMkytciNmmUo3NaNdC7t6IkRMlTZA/s16000/gboard_data.PNG" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Contents of Gboard's databases folder (v 10.0.02.338070508)</i></td></tr></tbody></table><p></p>In different versions of the app, the database formats and names have changed a bit. Of these, useful data can be found in trainingcache2.db, trainingcache3.db and trainingcachev2.db. Let's examine some of them now.<br /><div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif"><span style="font-size: 13px; white-space: pre-wrap;"><br /></span></span><h3 style="text-align: left;">trainingcache2.db (v 10.0.02.338070508)</h3>The table <b>training_input_events_table</b> contains information about the application in focus, its field name (where input was sent), the timestamp of event and a <a href="https://www.blogger.com/#">protobuf</a> BLOB stored in <b>_payload</b> field, as shown in screenshot below. <div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif"><span style="font-size: 13px; white-space: pre-wrap;"><br /></span></span></div><div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjvXanIb_fn5otSLwLbCw6G79TXqTU3VWE3sV21zgsjx7owCEeFnvolJej9C9ONC0Ti1vg3zBhj_ZbLrPWU74pGoLBj8xs5TXWiyJsX-AKhkCHb2_dZOAPeB6LB80MZMgACLwTGf6dTIc/s1135/v10.training_input_events_table-trainingcache2.db.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="605" data-original-width="1135" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjvXanIb_fn5otSLwLbCw6G79TXqTU3VWE3sV21zgsjx7owCEeFnvolJej9C9ONC0Ti1vg3zBhj_ZbLrPWU74pGoLBj8xs5TXWiyJsX-AKhkCHb2_dZOAPeB6LB80MZMgACLwTGf6dTIc/w640-h342/v10.training_input_events_table-trainingcache2.db.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - training_input_events_table (not all columns shown)</i></td></tr></tbody></table><br /></span>The highlighted entry above is from an app that was since deleted. The <b>_payload</b> BLOB is decoded in screenshot below, highlighting the text typed by the user in the Email input field. The protobuf has also has all of the data included in the other columns in the table.</div><div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif" style="font-size: 13px; white-space: pre-wrap;"><br /></span></div><div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHOGh2x3nVTjIxF-x4PvWePY0CSpXyrIPCzbt9_DHg31Z_7GoS8zSdxKoqvyFxAcYsdN9OCBQ0WGS95T4KzXbAEHlUJnMEMPpUjwfYR9kJxHrWPhiUprClZWTLNypmDwiWUR40e_a1SjQ/s338/cumberland+farms+blob.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="330" data-original-width="338" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHOGh2x3nVTjIxF-x4PvWePY0CSpXyrIPCzbt9_DHg31Z_7GoS8zSdxKoqvyFxAcYsdN9OCBQ0WGS95T4KzXbAEHlUJnMEMPpUjwfYR9kJxHrWPhiUprClZWTLNypmDwiWUR40e_a1SjQ/s320/cumberland+farms+blob.PNG" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - Decoded Protobuf from _payload column</i></td></tr></tbody></table><br /></span>In most instances however, the protobuf looks like this - see screenshot below, where input needs to be put back together as shown.. Here you can see the words the user typed as well as suggestions offered by the app. Suggestions can be for spelling, grammar, or contact names, or something else.</div><div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdgfUZdRE-NQ3HM3srGYUcb9_n-gjBhZdwJJt0ttnoD9irHWC5_boWof-pDAyUMB5PhR6i3k_Dz5_23HQtNgCelW2NehAbiBLNzvRL07aItMPkpY9mWMZtikMEnro4Xrk11LVGkbY9i0M/s611/training_input_events_table_blob_explained.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="611" data-original-width="444" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdgfUZdRE-NQ3HM3srGYUcb9_n-gjBhZdwJJt0ttnoD9irHWC5_boWof-pDAyUMB5PhR6i3k_Dz5_23HQtNgCelW2NehAbiBLNzvRL07aItMPkpY9mWMZtikMEnro4Xrk11LVGkbY9i0M/w466-h640/training_input_events_table_blob_explained.PNG" width="466" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 4 - Decoded protobuf - reconstructing user input</i></td></tr></tbody></table><br /></span>Above, you can see the words typed and suggestions offered. On an Android device, the suggestions appear as shown below while typing.</div><div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif"><span style="font-size: 13px; white-space: pre-wrap;"><br /></span></span></div><div><span face="Roboto, RobotoDraft, Helvetica, Arial, sans-serif"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2aabsn0GdbmK2D8yRRzjdACdYin039ynKP1FCjS5mPhiQk5vgXqv3_QC_DXOYDhfnCUIKL4F_tiGtESLnioiYpvqAoR2O8s-FeYkp5xGTIEjSh-heS4aw0SPtZ-tmetuwDDV0DoxpI-U/s534/suggesstions.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="449" data-original-width="534" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2aabsn0GdbmK2D8yRRzjdACdYin039ynKP1FCjS5mPhiQk5vgXqv3_QC_DXOYDhfnCUIKL4F_tiGtESLnioiYpvqAoR2O8s-FeYkp5xGTIEjSh-heS4aw0SPtZ-tmetuwDDV0DoxpI-U/s320/suggesstions.PNG" width="320" /></a></td></tr><tr><td class="tr-caption"><i>Figure 5 - Android keyboard highlighting suggested words</i></td></tr></tbody></table><br /></span><h3 style="text-align: left;">trainingcache3.db (v 10.0.02.338070508)</h3>In version 8.x, this same database is named <b>trainingcache2.db,</b> and follows the same exact format. The table <b>s_table</b> looks similar to the <b>training_input_events_table</b> seen earlier. However, the <b>_payload</b> field does not store the keystokes here.</div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtRDKlNvhTGJTYIDIYTv9osARZLpvi1Pq2dsoUqXL4vFepLMiey-_VZ0ORR6ja1IUp7C8Ijudpra3_0jK2sjLAS5aBdFutFcE_GiGE9-7izbH65KO2C9cUThqxmLb5vMDxlQ54I4qIy_g/s951/s_table.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="159" data-original-width="951" height="107" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtRDKlNvhTGJTYIDIYTv9osARZLpvi1Pq2dsoUqXL4vFepLMiey-_VZ0ORR6ja1IUp7C8Ijudpra3_0jK2sjLAS5aBdFutFcE_GiGE9-7izbH65KO2C9cUThqxmLb5vMDxlQ54I4qIy_g/w640-h107/s_table.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 6 - s_table</i></td></tr></tbody></table><br /><div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilEygQHggJ749RyJz0deuQj4A4Yv4iJaf7hAMDLc1mTZ_dh4wdD-0sv-v2DqkYjX6D5c-QtoqKSgHaL-1NhiNfZYfX6KZTBgoNBM7Rk2pavxq_tXyCDQcfyHgQ9UqeUpUKmwbz6tdB638/s263/s_table_pb.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="263" data-original-width="233" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilEygQHggJ749RyJz0deuQj4A4Yv4iJaf7hAMDLc1mTZ_dh4wdD-0sv-v2DqkYjX6D5c-QtoqKSgHaL-1NhiNfZYfX6KZTBgoNBM7Rk2pavxq_tXyCDQcfyHgQ9UqeUpUKmwbz6tdB638/s0/s_table_pb.PNG" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 7 - _payload protobuf decoded from s_table</i></td></tr></tbody></table><br /></div><div>Keystroke data is stored in the table <b>tf_table</b>. Here, most entries are a single key press, and to read this, it again needs to be put back together as shown below.</div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7Y1v12AsBa8W304A6thjMeREqqel4j0bYVMqWRSQd0k6CkB_iSts5EySG5Uh2NMMmrhGyHJ8za7ckjsrVepAsRgXZKYRs91MRy5kY_1NSyrt2Z_3qUVA9Zcut7tx7MT2xgl3TxfDKQgQ/s474/tf_table.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="474" data-original-width="426" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7Y1v12AsBa8W304A6thjMeREqqel4j0bYVMqWRSQd0k6CkB_iSts5EySG5Uh2NMMmrhGyHJ8za7ckjsrVepAsRgXZKYRs91MRy5kY_1NSyrt2Z_3qUVA9Zcut7tx7MT2xgl3TxfDKQgQ/w360-h400/tf_table.PNG" width="360" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 8 - tf_table entries</i></td></tr></tbody></table><br /></div><div>All keystrokes from the same session have the same <b>f1</b> value (a timestamp like field but not used as a timestamp). The order of the keys pressed is stored in <b>f4</b>. Assuming they are all in order, we can run a short query to concatenate the <b>f3</b> column values for easy reading (shown below). This isn't perfect, as group_concat() doesn't guarantee order of concatenation, but it seems to work for now!<div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvbG-rHIhGaax7C0YEmE_58Sw3S7C6Pv_twGZ3Wc09EATMC3XYTkIj4ZDFpJYUx-AKNUCLS9kEl6iqPKC8L1M7i2g4_0y1nW18F1Kv6gPytUaOWKv4YFbN-_CwHA0wQYSfSlmf1h4T9uo/s597/josh_query.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="422" data-original-width="597" height="453" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvbG-rHIhGaax7C0YEmE_58Sw3S7C6Pv_twGZ3Wc09EATMC3XYTkIj4ZDFpJYUx-AKNUCLS9kEl6iqPKC8L1M7i2g4_0y1nW18F1Kv6gPytUaOWKv4YFbN-_CwHA0wQYSfSlmf1h4T9uo/w640-h453/josh_query.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 9 - Reading keystroke sessions from tf_table</i></td></tr></tbody></table><br />We can combine (join) this data with the one from <b>s_table</b> to recreate the same data as we got from <b>training_input_events_table</b> earlier. </div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBPGKz27B18s-kyCjQf8BehGnpojT-vmHEts_4BKe782LOEPmOlfKG5z7zb0Ka3bTyCXEv7a-tQVn401pHBb9Kz1KRTPBPMDIbW67teNGQZdesSp8tw6mg5wbJvIaxcpv6r6ckxX_YfDY/s893/joined_tables_data.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="544" data-original-width="893" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBPGKz27B18s-kyCjQf8BehGnpojT-vmHEts_4BKe782LOEPmOlfKG5z7zb0Ka3bTyCXEv7a-tQVn401pHBb9Kz1KRTPBPMDIbW67teNGQZdesSp8tw6mg5wbJvIaxcpv6r6ckxX_YfDY/w640-h390/joined_tables_data.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 10 - joined tables</i></td></tr></tbody></table><br /><div>In the screenshot shown above, you can even see data being typed into a google doc, not saved locally. Only a snippet is shown above, but if you want to see the full parsed data, get <a href="https://thebinaryhick.blog/2020/02/15/android-10-image-now-available/" target="_blank">Josh's Android image(s)</a>, and the latest version of <a href="https://github.com/abrignoni/ALEAPP" target="_blank">ALEAPP</a> (code), which now parses this out. Below is a preview (from a different image my students might recognize).</div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLc_IaBNYUffU0pUrXorlhgfbUpVmqnpPGF44XbITnTP4rhPn30y-dd0Ujxn3fhxVwvXg1uNoLb6300H1_VCDYDfUmTrqm8RGF7TGLnBnJRZrMc05jVCGJvDLogut0a81DPEiXyr7OrIU/s1172/aleapp_trainingcache.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="432" data-original-width="1172" height="237" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLc_IaBNYUffU0pUrXorlhgfbUpVmqnpPGF44XbITnTP4rhPn30y-dd0Ujxn3fhxVwvXg1uNoLb6300H1_VCDYDfUmTrqm8RGF7TGLnBnJRZrMc05jVCGJvDLogut0a81DPEiXyr7OrIU/w640-h237/aleapp_trainingcache.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 11 - ALEAPP output showing trainingcache parsed output</i></td></tr></tbody></table><br /><div><div><br /></div><div>Cached keystroke data can also be seen and reconstructed from <b>trainingcachev2.db</b>, whose format is a bit different (not discussed here). Nothing of significance was found in trainingcache4 or the other databases. </div><div><br /><h3 style="text-align: left;">Observations</h3>As expected, keystrokes from password fields are not stored or tracked. <br /><br /></div><div>In data reconstructed from tf_table, you can see all the spelling mistakes a user made while typing! Any corrections made in the middle of a word/sentence will be seen at the end (because we are getting the raw keystokes in order of keys pressed). Hence it might be difficult to read some input. Also, if a user types something into a field, then deletes a word(s), and retypes, you won't see the final edited (clean) version, as backspaces (delete) are not tracked. You can see some of this in the output above (figure 9).</div></div></div><div><br /></div><div>The caches are periodically deleted (and likely size limited too), and so you shouldn't expect to find all user typed data here. </div><div><br /></div>Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-33004476766803003662021-01-03T11:59:00.011-05:002021-01-03T12:05:51.422-05:00iOS Application Groups & Shared data<h3 style="text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8Lm3RNFJ1XqiHlcFwnhY-QGj2Z_fxUHKAsCDYGSbPMiQhBk-STVS-JkcqGIs5SMPUN0TjHPbY0zsoACAh9k-Rb3CNlVXjnlOUzb1smvR3b5PZsoArd532VKNAq7-BMxrorH_CNKW7M6k/s512/660ma652j22kbh422716akluv6.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="512" data-original-width="512" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8Lm3RNFJ1XqiHlcFwnhY-QGj2Z_fxUHKAsCDYGSbPMiQhBk-STVS-JkcqGIs5SMPUN0TjHPbY0zsoACAh9k-Rb3CNlVXjnlOUzb1smvR3b5PZsoArd532VKNAq7-BMxrorH_CNKW7M6k/w200-h200/660ma652j22kbh422716akluv6.png" width="200" /></a></div>Background<br /></h3><p>Tracking down an iOS application's Data folder, aka, SandboxPath in iOS is fairly easy. One simply needs to look at the <b>applicationState.db sqlite</b> database located under <b>/private/var/mobile/Library/FrontBoard/</b> This is well known. </p><p>However locating the sandbox folder for its AppGroups (and Extensions) is not so straight-forward. The suggested method by Scott Vance <a href="https://blog.d204n6.com/2020/09/ios-tracking-bundle-ids-for-containers.html" target="_blank">here</a>, and recommended by few others too is to look for the <b>.com.apple.mobile_container_manager.metadata.plist</b> file under each of the UUID folders:</p><p></p><ul style="text-align: left;"><li>/private/var/containers/Shared/SystemGroup/<span style="color: red;">UUID</span>/</li><li>/private/var/mobile/Containers/Shared/AppGroup/<span style="color: red;">UUID</span>/</li><li>/private/var/mobile/Containers/Data/InternalDaemon/<span style="color: red;">UUID</span>/</li><li>/private/var/mobile/Containers/Data/PluginKitPlugin/<span style="color: red;">UUID</span>/</li></ul><p></p><p>As noted by Scott, the <a href="https://github.com/abrignoni/iLEAPP" target="_blank">iLEAPP</a> tool does this too, reading all the plists and listing out the path and its group name. For manual analysis, this works out great, as you can visually make out the app name from the group name. For example, the Notes app has bundle_id <b>com.apple.mobilenotes</b> and one of its shared groups (where the actual Notes db is stored!) has the id <b>group.com.apple.notes</b>.</p><h3 style="text-align: left;">The Problem</h3><p>For automated analysis, this approach does not work, as each app follows its own convention on naming for ids. A program cannot know that <b>group.com.apple.notes</b> corresponds to <b>com.apple.mobilenotes</b>. Hence we search for something with a more direct reference connecting Shared Containers to their Apps. Before we proceed further, its important to understand the relationships between extensions, apps and shared containers. The diagram below does a good job of summarizing this. The shared containers are identified by AppGroups.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2a0kw7PI4rnj-by28I6yqy7_9M5EloqIoA_4vUHdF8LvqBfXBgSkV7y9F4BuVR8bY-kUfDyTP9BL_00SG8IJXD4B-otQbB0eh2CILN_qzkic2ppmWc54qd7I4h9veTAAGLoPuOkNj_Co/s1176/bundle_ext_shared.jpeg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="510" data-original-width="1176" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2a0kw7PI4rnj-by28I6yqy7_9M5EloqIoA_4vUHdF8LvqBfXBgSkV7y9F4BuVR8bY-kUfDyTP9BL_00SG8IJXD4B-otQbB0eh2CILN_qzkic2ppmWc54qd7I4h9veTAAGLoPuOkNj_Co/w640-h277/bundle_ext_shared.jpeg" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - iOS App, Extension, container relationships - Source: <a href="https://medium.com/@manibatra23/sharing-data-using-core-data-ios-app-and-extension-fb0a176eaee9" target="_blank">https://medium.com/@manibatra23/sharing-data-using-core-data-ios-app-and-extension-fb0a176eaee9</a></i></td></tr></tbody></table><p><br /></p><h3 style="text-align: left;">The Solution</h3><p>Fortunately, there is a database that tracks container information on iOS. It is located at <span style="background-color: #fcff01;">/private/var/root/Library/MobileContainerManager/containers.sqlite3</span></p><p>It precisely lists all Apps, their extensions, AppGroups and Entitlements. As far as I can tell, this is the only place where this information is stored (apart from caches and logs). It does not have information about UUIDs. This database is listed in the SANS smartphone forensics poster, but I couldn't find any details on it elsewhere. </p><p>The database structure is simple with just 3 main tables (and an sqlite_sequence one). </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5xdEzQFpkTJCHMkalLf6haxWaFc0Z_w4VWJgacigLp0oCroccM45sHT4ibIt8EiOZz6M37VZhDzS5bVU-qtbz8o1YxIT6mH0iO2r-p4QihGQlgrFpX9bmC2TOg6gOXrN9_6M5m1IRJxc/s195/containers_db.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="187" data-original-width="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5xdEzQFpkTJCHMkalLf6haxWaFc0Z_w4VWJgacigLp0oCroccM45sHT4ibIt8EiOZz6M37VZhDzS5bVU-qtbz8o1YxIT6mH0iO2r-p4QihGQlgrFpX9bmC2TOg6gOXrN9_6M5m1IRJxc/s0/containers_db.PNG" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i><br />Figure 2 - containers.sqlite database tables</i></td></tr></tbody></table><br />The <b>child_bundles </b>table lists extensions and their owner Apps. In figure below, you can see the extensions for the <b>com.apple.mobilenotes</b> app.<div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgn0FwIhzokjon92EJo_Fhmob3ka-bdVzN3JdymlB2itGbjYXsauLpsHe1XBMtMKDIdG2mZaYCmqHbIOVugY3xllhIzhBNNDkllpEdYuipvMVj3ZdyBvESICehmkRUOxTFUXJ5CyWGWVbI/s433/notes_and_ext.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="225" data-original-width="433" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgn0FwIhzokjon92EJo_Fhmob3ka-bdVzN3JdymlB2itGbjYXsauLpsHe1XBMtMKDIdG2mZaYCmqHbIOVugY3xllhIzhBNNDkllpEdYuipvMVj3ZdyBvESICehmkRUOxTFUXJ5CyWGWVbI/s16000/notes_and_ext.PNG" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - child_bundles table, filtered on 'notes'</i></td></tr></tbody></table><br />Or one could write a small query to list all apps with their extension names like shown below.</div><div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-WPFTXlauDd2YQLi9rlNfVyfHepnpDMe8MQgytQc_lSKUYtZuPDCNmBDroGaENmVg1ECq7Yasbbf4LosDDy_00BJ1z5PSIcmglFh0k1VBRTcH5fySZ1N_MltYoLrmcrlsSuRRnaYjRcE/s751/query_output.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="433" data-original-width="751" height="370" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-WPFTXlauDd2YQLi9rlNfVyfHepnpDMe8MQgytQc_lSKUYtZuPDCNmBDroGaENmVg1ECq7Yasbbf4LosDDy_00BJ1z5PSIcmglFh0k1VBRTcH5fySZ1N_MltYoLrmcrlsSuRRnaYjRcE/w640-h370/query_output.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 4 - App & Extensions - query and output</i></td></tr></tbody></table><br /><br />Information about AppGroups is found in the data field of the <b>code_signing_data </b>table as a BLOB, which stores a binary plist.</div><div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdfBXHYrkeV_WoCbwzedfDOtky89epRaHTCrDITLcpYtwh0IVPxXBdH4P3fpztckDu23R7QY_YTmA6d9a1AzxAlLVQfCJiU-Kx38CKb_Eno5CyteabSXtcQiizBWoL3zndX9di4e-09rg/s907/blob_and_plist.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="219" data-original-width="907" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdfBXHYrkeV_WoCbwzedfDOtky89epRaHTCrDITLcpYtwh0IVPxXBdH4P3fpztckDu23R7QY_YTmA6d9a1AzxAlLVQfCJiU-Kx38CKb_Eno5CyteabSXtcQiizBWoL3zndX9di4e-09rg/s16000/blob_and_plist.PNG" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 5 - Plist (for com.apple.mobilenotes - cs_info_id 456) from 'code_signing_data.data'</i></td></tr></tbody></table><br /><p>The Entitlements dictionary has a lot of information in it. If this App creates a shared AppGroup, then it will show up under <b>com.apple.security.application-groups</b>. There may also be groups under <b>com.apple.security.system-groups</b>.<br /></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjicYDz8nZG8F9uZz3-HSkBJD_lA9vOD23sbK000voEjJr0IN4fXqDnkHBuHj9hCR3sqTyDVgXugdNGRC0BV1LWCT_k_DzCpnXkyJPrkcGMAsE2H-O5PwHoyVEgLVvZibmxiyaifBDiNyU/s639/group_names_in_plist.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="149" data-original-width="639" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjicYDz8nZG8F9uZz3-HSkBJD_lA9vOD23sbK000voEjJr0IN4fXqDnkHBuHj9hCR3sqTyDVgXugdNGRC0BV1LWCT_k_DzCpnXkyJPrkcGMAsE2H-O5PwHoyVEgLVvZibmxiyaifBDiNyU/s16000/group_names_in_plist.PNG" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 6 - AppGroup information in Entitlements section (in plist)</i></td></tr></tbody></table><br /><p>So from the above data, we know that the Notes App has 5 <i>extensions </i>and 2 <i>AppGroups</i>, and we have the exact string names(aka ids) too - <b>group.com.apple.notes</b> and <b>group.com.apple.notes.import</b> . Correlating this data with information we found from <b>.com.apple.mobile_container_manager.metadata.plist</b> files (from each UUID folder earlier), we can programmatically search and link the two as being part of the same App, based on the container id (AppGroup name).</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWS-xfI0tir5e5RGJAhFGqje_pN-P5xFulqcowV7hN2iTcmccnPWHtsJYlbsS5OZtfJstEASN09_7ERqkbsQZ75CZ_6Rmdr0EMrVK-eVKajdcjWleQFOf82WF43ABeU3TAzZuY7CljNJU/s643/mcm_plist_folder.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="453" data-original-width="643" height="450" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWS-xfI0tir5e5RGJAhFGqje_pN-P5xFulqcowV7hN2iTcmccnPWHtsJYlbsS5OZtfJstEASN09_7ERqkbsQZ75CZ_6Rmdr0EMrVK-eVKajdcjWleQFOf82WF43ABeU3TAzZuY7CljNJU/w640-h450/mcm_plist_folder.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 7 - AppGroup/UUID folder showing plist's content and Container owner id</i></td></tr></tbody></table><br /><p><span style="font-size: medium;">This methodology is implemented in the APPS plugin for <a href="https://github.com/ydkhatri/mac_apt" target="_blank">ios_apt</a>, which now lists every App, it's AppGroups, SystemGroups, Extensions, and all the relationships. So you don't have to do any of it manually now. Enjoy!</span></p><div class="separator" style="clear: both; text-align: center;"><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKMqx3t73j2k1xcjhcuuhiMeLEalHFUBt5Yrf84mGoj891KT2KQ3p-eyMGHYmTPCt5xgH9-0kz9GO8SkNFlLodHTLI0ZoX6_zJoBWT4gNMgxg4MaGxS7ru1xpYXGpYt_14nOAwtd4V55k/s943/apps_appgroups.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="305" data-original-width="943" height="207" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKMqx3t73j2k1xcjhcuuhiMeLEalHFUBt5Yrf84mGoj891KT2KQ3p-eyMGHYmTPCt5xgH9-0kz9GO8SkNFlLodHTLI0ZoX6_zJoBWT4gNMgxg4MaGxS7ru1xpYXGpYt_14nOAwtd4V55k/w640-h207/apps_appgroups.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Figure 8 - Apps Table from ios_apt output (not all columns are shown here)</i></td></tr></tbody></table></div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1PN3iHVTACq9BFtlKnQ8S7ToxaU5Ogzubuy78MzC_O5OSrfVga8-LRw1JLhRszwiKQ7mDpX_vPWMDJbLl2g_TXyFnDXxfX4UKFvqlrN2uPRET4FoogLPYTPwxh-XV3bZQOo3q1P0U6QE/s1228/AppGroupInfo.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="339" data-original-width="1228" height="176" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1PN3iHVTACq9BFtlKnQ8S7ToxaU5Ogzubuy78MzC_O5OSrfVga8-LRw1JLhRszwiKQ7mDpX_vPWMDJbLl2g_TXyFnDXxfX4UKFvqlrN2uPRET4FoogLPYTPwxh-XV3bZQOo3q1P0U6QE/w640-h176/AppGroupInfo.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><div class="separator" style="clear: both;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td class="tr-caption" style="text-align: center;"><i>Figure 9 - AppGroupInfo Table from ios_apt output<br /><br /></i></td></tr></tbody></table></div></td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"><br /></div></div>Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com1tag:blogger.com,1999:blog-1264611260322778486.post-59196518581980607532020-12-28T22:04:00.000-05:002020-12-28T22:07:21.407-05:00Introducing ios_apt - iOS Artifact Parsing Tool<h3 style="text-align: left;"><div style="text-align: right;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqGE2u1pnjf6bhNnS3LoU_iL-zAm1Iz4faWJPF5iScFAGDzdzlD1m0VsAa_SjUrmXhfhXMuDHTrVR1JpOTQ0FBfOA4vH7Oi0fzOoPjzx2tMvaTNKF1JOZUbV8R655TrSU6O5L0WJJYTRA/s1600/ios.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1600" data-original-width="1600" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqGE2u1pnjf6bhNnS3LoU_iL-zAm1Iz4faWJPF5iScFAGDzdzlD1m0VsAa_SjUrmXhfhXMuDHTrVR1JpOTQ0FBfOA4vH7Oi0fzOoPjzx2tMvaTNKF1JOZUbV8R655TrSU6O5L0WJJYTRA/w200-h200/ios.png" width="200" /></a></div>ios_apt is the new shiny companion to <a href="https://github.com/ydkhatri/mac_apt">mac_apt<br /></a></h3><p>ios_apt is not a separate project, it's just a part of the mac_apt framework, and serves as a launch script that processes iOS/iPadOS artifacts. </p><h3 style="text-align: left;">Why yet another iOS parsing tool, don't we already have too many?</h3><p>In addition to paid tools, we have <a href="https://github.com/abrignoni/iLEAPP" target="_blank">iLEAPP</a>, <a href="https://github.com/mac4n6/APOLLO" target="_blank">APOLLO</a> and a few others, and I am also an active contributor to some of them. This isn't meant to compete with them, rather it utilizes the mac_apt framework to prevent duplication of work. </p><p>Many artifacts on iOS and macOS share common backend databases, configuration and artifact types. Among the artifacts that are almost identical are -</p><p></p><ul style="text-align: left;"><li>Spotlight</li><li>UnifiedLogging logs</li><li>Network usage database</li><li>Networking artifacts like hardware info and last IP leases</li><li>Safari</li><li>Notes</li><li>FSevents</li><li>ScreenTime</li></ul><p></p><p>There are a few others too that aren't listed here. But you get the picture. Since mac_apt already parsed all of them, it made sense to just create an ios variant that parses these from ios extractions. </p><p>Also many of these artifacts are fairly complex and other FOSS tools don't have the architecture needed to handle them. APOLLO only gathers information from SQLite databases. iLEAPP is geared towards single artifact parsing per plugin. It is not designed for multiple layers of parsing where information parsed from one artifact/file may be used as a key to jump to an artifact elsewhere on disk.</p><h3 style="text-align: left;">Limitations</h3><p>In its first version, ios_apt only works on full file system images extracted out to a folder. No support yet for zip/tar/dar/7z/other archives.</p><h3 style="text-align: left;">Available Plugins / Modules</h3><p>The following Plugins are available as of now -</p><p></p><ul style="text-align: left;"><li>APPS</li><li>BASICINFO</li><li>FSEVENTS</li><li>NETUSAGE</li><li>NETWORKING</li><li>NOTES</li><li>SAFARI</li><li>SCREENTIME</li><li>SPOTLIGHT</li><li>TERMSESSIONS</li><li>WIFI</li></ul><div><i><b>Download the latest version of <a href="https://github.com/ydkhatri/mac_apt/releases/" target="_blank">mac_apt</a> to get ios_apt.</b></i></div><p></p>Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-74260320793232085002020-07-19T22:25:00.000-04:002020-07-19T22:29:26.795-04:00KTX to PNG in Python for iOS snapshots<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqJkDUNelnTAWhuGzXHrPMaEv5azlTHL3C8S8sjCIvVEIQtC1atlZU85X8DPtCr4zD_2MBe3RpzWAWQ2eOhpVSDpgW59BTrJa50tsdcmm_b_YQ_3Rl-VXMwwkQ7CKzj-7agEu_4qwn20s/s1600/ktx2png.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="165" data-original-width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqJkDUNelnTAWhuGzXHrPMaEv5azlTHL3C8S8sjCIvVEIQtC1atlZU85X8DPtCr4zD_2MBe3RpzWAWQ2eOhpVSDpgW59BTrJa50tsdcmm_b_YQ_3Rl-VXMwwkQ7CKzj-7agEu_4qwn20s/s1600/ktx2png.PNG" /></a></div>
App snapshots on iOS are stored as KTX files, this is fairly well known at this point, thanks to the research by Geraldine Blay (<a href="https://twitter.com/@i_am_the_gia" target="_blank">@i_am_the_gia</a>) and Alex Brignoni (<a href="https://twitter.com/AlexisBrignoni" target="_blank">@AlexisBrignoni</a>) <a href="https://gforce4n6.blogspot.com/2019/09/a-quick-look-into-ios-snapshots.html" target="_blank">here</a> and <a href="https://abrignoni.blogspot.com/2019/09/ios-snapshots-triage-parser-working.html" target="_blank">here</a>. They even came up with a way to collect and convert them to PNG format. However that solution was only for macOS, and hence this research..<br />
<br />
<h3 style="text-align: left;">
KTX </h3>
KTX is a file format used to store textures, and used commonly by OpenGL programs. The KTX file format is known and available <a href="https://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/" target="_blank">here</a>. There aren't many standalone utilities that work with KTX files, as it is mostly used in games and not for reading/distributing standalone textures. There are also no readily available python libraries to read it! The Khronos group that created the format distributes libktx, but it is C++ only. Even so, it would not be able to read iOS created ktx files (read on for the reasons mentioned below). The few Windows applications I could find like <a href="https://pixelandpolygon.com/" target="_blank">PicoPixel </a>would not recognize Apple created KTX files as valid files.<br />
<br />
So what is different here? A quick glance over the file in the hex editor showed that the texture data was stored in LZFSE compressed form, which currently only macOS/iOS can read.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsHFK8n1G3q8f6Jtd41LEZvrSimNTa8hbzoMYTk1wYVYrQdGSe4e49lzJb9y6kG-GgQBK5VNr-Xacam81qvfxBQnyXCr1KfA7Dh80I08qdYEfr4GmVN4dJZ6x5-ZXvja7j7__u6UHk0dc/s1600/ktx_hex.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="427" data-original-width="646" height="264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsHFK8n1G3q8f6Jtd41LEZvrSimNTa8hbzoMYTk1wYVYrQdGSe4e49lzJb9y6kG-GgQBK5VNr-Xacam81qvfxBQnyXCr1KfA7Dh80I08qdYEfr4GmVN4dJZ6x5-ZXvja7j7__u6UHk0dc/s400/ktx_hex.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure - Ascii view of 010 hex editor with ktx template</i></td></tr>
</tbody></table>
<br />
Now using <a href="https://github.com/ydkhatri/pyliblzfse" target="_blank">pyliblzfse</a>, I could decompress the data, and recreate a new KTX file with raw texture data. Even so, it would not render with KTX viewers other than macOS's Finder/Quickview and Preview. So I tried a few different approaches to get to the data.<br />
<br />
<h4 style="text-align: left;">
Attempt 1 - Rendering & Export</h4>
Textures are different from 2D images and there is hence not a direct conversion from a textures to an image format. From available research, it seemed like the easiest way to do this would be to use OpenGL to render the texture data (extracted from the KTX file), then use OpenGL to save a 2D image of the rendered image. There is some sample code available too on the internet, but in order to get it to work, one would need to know how to use OpenGL to render textures, a learning curve that was too steep for me..<br />
<br />
After spending several hours trying to get this to work in Python, I ultimately gave up as python is not the platform where major OpenGL development takes place, therefore there is little to no support, and libraries are platform dependent. I barely got one of the libraries to install correctly in Linux, and every step of the way I got more errors than I wanted to debug, ultimately I threw in the towel.<br />
<br />
<h4 style="text-align: left;">
Attempt 2 - Convert texture data to RAW image data</h4>
Reading the KTX file header, the <i><b>glInternalFormat </b></i>value field from the header is <i><b>0x93B0 </b></i>for all iOS produced KTX files (as seen in screenshot above). This value is the enumeration for <i><b>COMPRESSED_RGBA_ASTC_4x4</b></i>. So now we know the format is ASTC, which is Adaptive Scalable Texture Compression, a lossy compressed format for storing texture data, and uses a block size of 4x4 pixels. That simplifies our task to now finding a way to convert ASTC data to raw image data. A bit of searching led me to the python library <i><b>astc_decomp </b></i>which does precisely that. So what I needed now was to put the pieces together as follows:<br />
<br />
<ol style="text-align: left;">
<li>Read KTX file and parse format to get LZFSE compressed data, and parameters of Width and Height</li>
<li>Decompress LZFSE to get ASTC data</li>
<li>Convert ASTC to RAW image stream</li>
<li>Save RAW image as PNG using PIL library</li>
</ol>
<br />
Combining this together, we are able to create a python script that can convert KTX files to PNG files. Get it here:<br />
<a href="https://github.com/ydkhatri/MacForensics/tree/master/IOS_KTX_TO_PNG">https://github.com/ydkhatri/MacForensics/tree/master/IOS_KTX_TO_PNG</a><br />
<br />
There is also a windows compiled executable there if you need to do this on windows without python. Alex Brignoni was helpful in sending me samples of KTX files to work with from multiple images. The code also works with KTX files that are not really KTX, ie, they have the .ktx extension but the header is 'AAPL'. The format is however similar and my code will parse them out too. If you do come across a file that does not work, send it to me and I can take a look.<br />
<br />
A point to note is that not all KTX files use the <i>COMPRESSED_RGBA_ASTC_4x4</i> format, only the iOS created ones do. So you may come across many KTX files deployed or shipped with apps that can't be parsed with this tool, as it only handles ASTC 4x4 format.<br />
<br />
Enjoy!</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com1tag:blogger.com,1999:blog-1264611260322778486.post-59373144150345149302020-06-09T23:05:00.001-04:002020-06-09T23:05:27.471-04:00Screentime Notifications in Catalina (10.15)<div dir="ltr" style="text-align: left;" trbidi="on">
If you routinely perform mac forensics, you've probably done a few macOS Catalina (10.15) examinations already. And if you are the kind that verifies your data, you may have noticed that for ScreenTime notifications the databases don't show you the same strings that you see in the actual displayed Notification and several forensic tools don't either.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkO6LM1XEFwBW9QCISiMmlxcIp6t0VL-Z4kn5QXHZbtUphdsVQzXw0SfAB_l9k2FxJODIVkGLMqkeD8NlYjv7cAQaWNZM3o0iN_Yg7Oc94MBGYXOWZGQscFQPVw_c9DkLOkz4JmwEj00Y/s1600/notifications.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="204" data-original-width="664" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkO6LM1XEFwBW9QCISiMmlxcIp6t0VL-Z4kn5QXHZbtUphdsVQzXw0SfAB_l9k2FxJODIVkGLMqkeD8NlYjv7cAQaWNZM3o0iN_Yg7Oc94MBGYXOWZGQscFQPVw_c9DkLOkz4JmwEj00Y/s1600/notifications.png" /></a></div>
<br />
Let's explore why.<br />
<br />
To start with, lets review the format of the Notifications database. For macOS High Sierra (10.) and above, it is located at:<br />
<br />
<span style="background-color: yellow;">/private/var/folders/<span style="color: red;"><xx>/<yyyyyyy></span>/0/com.apple.notificationcenter/db2/db</span><br />
<br />
where the <span style="color: red;"><b><xx>/<yyyyyy></b></span> portion represents what might appear like random strings, but they are not random. This folder path represents the <b>DARWIN_USER_DIR</b> for a specific user. For more details on this read my old post <a href="https://www.swiftforensics.com/2017/04/the-mystery-of-varfolders-on-osx.html" target="_blank">here</a>.<br />
<br />
Inside the database, the <b>record</b> table holds the actual notification data (title, sub-title, body) and date of notification among other fields. A simple database query can get the useful data.<br />
<br />
<b><span style="font-family: "courier new" , "courier" , monospace;"><span style="color: blue;">SELECT</span> </span></b><br />
<b><span style="font-family: "courier new" , "courier" , monospace;"> (<span style="color: blue;">SELECT</span> identifier <span style="color: blue;">FROM</span> app <span style="color: blue;">WHERE</span> app.app_id=record.app_id) <span style="color: blue;">as</span> app,</span></b><br />
<b><span style="font-family: "courier new" , "courier" , monospace;"> uuid, data, presented, delivered_date</span></b><br />
<b><span style="font-family: "courier new" , "courier" , monospace;"><span style="color: blue;">FROM</span> record</span></b><br />
<br />
The actual notification data is within a plist stored in the column <b>data</b>. Inside this plist, you can easily navigate to the items <b>titl</b>, <b>subt</b> and <b>body </b>to get the title, sub-title and body. However for screentime notifications, the data looks different. Instead of individual strings in these values, they are lists.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJuy2QI3hCcrficn0Pm1cLVytNfSYjl2eagpwSF0prl9oF_ioIY5lHPEYEyB7FvVV846Kq_0t-BXpf4P7dWAnrVdHJjdWppDE-gt1hoQSH_OP0aNyk7oLJgYLGVwRpjUN5kHt65Q3OO2U/s1600/screentime_notif_plist.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="732" data-original-width="946" height="495" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJuy2QI3hCcrficn0Pm1cLVytNfSYjl2eagpwSF0prl9oF_ioIY5lHPEYEyB7FvVV846Kq_0t-BXpf4P7dWAnrVdHJjdWppDE-gt1hoQSH_OP0aNyk7oLJgYLGVwRpjUN5kHt65Q3OO2U/s640/screentime_notif_plist.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Embedded plist for screentime notification</i></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: left;">
Screentime uses format strings and a list of data, which needs to be put back together. This is similar to how Event logs in windows or Unified logging in macOS works. The format strings are located at the paths shown below (for english) and are available in other languages too:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<span style="background-color: yellow;"><span style="font-size: x-small;">/System/Library/UserNotifications/Bundles/com.apple.ScreenTimeNotifications.bundle/Contents/Resources/en.lproj/Localizable.strings</span></span><br />
<span style="background-color: yellow;"><span style="font-size: x-small;">/System/Library/UserNotifications/Bundles/com.apple.ScreenTimeNotifications.bundle/Contents/Resources/en.lproj/InfoPlist.strings</span></span><br />
<br />
These files are plists which consist of a single dictionary each. So <i>WeeklyReportNotificationNegativeDeltaBody</i> seen in plist above resolves to the message :<br />
<i style="font-family: Georgia, "Times New Roman", serif;">"Your screen time was down %@ last week, for an average of %@ a day." </i><span style="font-family: inherit;">The <b>%@ </b>will be replaced with data provided (<i>15%</i> and <i>6 hours, 24 minutes</i>) becoming:</span><br />
<i style="font-family: Georgia, "Times New Roman", serif;">"Your screen time was down 15% last week, for an average of 6 hours, 24 minutes a day."</i><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSL7p4JXRZyJhn9_Hj0Q4LDQdzDhsjlHICmaC6mFDUD_Ve2cRFl9Tc39QfyPboz2bGldRjpt3JllyGpRy13XZ55kxtbXnoYNMeraCJtli1-NZAoD9IinHiA-vuxEpAV_wKxiEbveYakug/s1600/Localizable.strings.snippet.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="123" data-original-width="829" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSL7p4JXRZyJhn9_Hj0Q4LDQdzDhsjlHICmaC6mFDUD_Ve2cRFl9Tc39QfyPboz2bGldRjpt3JllyGpRy13XZ55kxtbXnoYNMeraCJtli1-NZAoD9IinHiA-vuxEpAV_wKxiEbveYakug/s1600/Localizable.strings.snippet.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - Snippet of Localizable.strings plist</i></td></tr>
</tbody></table>
Similarly <i>WeeklyReportNotificationTitle</i> becomes <i>Weekly Report Available</i>. So now, we are able to reconstruct the complete original message.<br />
<br />
<a href="https://github.com/ydkhatri/mac_apt" target="_blank">mac_apt's</a> NOTIFICATIONS plugin has now been updated with this functionality.</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-555658641745003642020-03-30T23:19:00.004-04:002020-08-01T01:59:05.013-04:00 Parsing unknown protobufs with python<div dir="ltr" style="text-align: left;" trbidi="on">
Protocol Buffers are quite popular, more and more apps and system files are storing data in this format in both iOS and Android operating systems. If you aren't familiar with Protocol Buffers, read this <a href="https://www.swiftforensics.com/2020/01/usagestats-on-android-10-q.html" target="_blank">post</a>. There I use the <i>protoc.exe</i> utility (by google), as does everyone else who needs to view this data, when you do not have the corresponding <span style="font-family: "courier new" , "courier" , monospace;">.proto</span> file.<br />
<div>
<br /></div>
<div>
This is great! But the <b>raw </b>view/output has one big disadvantage. While this approach (<span style="font-family: "courier new" , "courier" , monospace;">--decode_raw</span>) works fine if you just want to see the text strings stored in your data, it does not always provide the correct conversions for all the raw data types! </div>
<div>
<br /></div>
<div>
According to <a href="https://developers.google.com/protocol-buffers/docs/encoding#structure" target="_blank">google</a>, when the message (data) is encoded, there are only 6 different types of data types allowed. These are known as <b>wire types</b>. Here are the allowed types (below).</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-HxCFnbyqN5m6dMZUvHYQvpr7X1NJXR3P7wWH7JGbUcmiEbaMThyphenhyphen8JEN1utcdrre7k_YkRRncH3F1KjcOW42C7gtoFmWOZq9pKz71kyh8rUY186qgwpkCVHoAYI1YCIw1Lzh3iob1VYM/s1600/wiretypes_pb.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="274" data-original-width="662" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-HxCFnbyqN5m6dMZUvHYQvpr7X1NJXR3P7wWH7JGbUcmiEbaMThyphenhyphen8JEN1utcdrre7k_YkRRncH3F1KjcOW42C7gtoFmWOZq9pKz71kyh8rUY186qgwpkCVHoAYI1YCIw1Lzh3iob1VYM/s1600/wiretypes_pb.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure - Allowed wire types from https://developers.google.com/protocol-buffers/docs/encoding#structure</i></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Unless you have the <span style="font-family: "courier new" , "courier" , monospace;">.proto</span> file, you really don't know what the original data type may be. Even <i>protoc.exe</i> just makes a best guess. For instance, all binary blobs are also converted to strings with <i>protoc </i>as both the <i>string </i>and <i>bytes </i>type<i> </i>use the<i> Length-delimited</i> wiretype. There is also no way to tell if a number is to be interpreted as signed or unsigned, because they both use the same underlying type (<i>varint</i>)!</div>
<div>
<br /></div>
<div>
Now to raw-decode a protobuf in python, there are a couple of libraries I have seen so far that do a decent job. I will list out the libraries, then demonstrate parsing with them, and compare.</div>
<div>
<br /></div>
<div>
<span style="font-size: large;">1. Protobuf-decoder</span> (<a href="https://github.com/nevermoe/protobuf-decoder">https://github.com/nevermoe/protobuf-decoder</a>)</div>
<div>
This seems to be more than 4 years old and not maintained any more. It is also in python2. There is a python3 port somewhere. It makes several assumptions regarding data types and attempts to produce output similar to <i>protoc</i>.<br />
<br /></div>
<div>
</div>
<div>
<span style="font-size: large;">2. BlackboxProtobuf </span>(<a href="https://github.com/nccgroup/blackboxprotobuf">https://github.com/nccgroup/blackboxprotobuf</a>)</div>
<div>
This is a more mature library that provides much more in functionality. It makes relatively few assumptions about data types. In addition to parsing the protobuf and returning a dictionary object, it also provides a type definition dictionary for the parsed data. </div>
<div>
<br />
To demonstrate what I am talking about, I created a demo protocol buffer file called <i>addressbook.proto</i> and defined a protobuf message as shown below.</div>
<div>
<br /></div>
<div>
<div style="background: #eeeeee; font-family: "courier new" , "courier" , monospace;">
message Person {<br />
required string name = 1;<br />
required int32 id = 2;<br />
optional string email = 3;<br />
required int64 id64 = 4;<br />
required uint64 uid64 = 5;<br />
optional double double = 6;<br />
optional bytes bytes = 7;<br />
}</div>
</div>
<div>
<br /></div>
<div>
Then compiled it using <i>protoc.exe</i>.<br />
<br /></div>
<div>
<span style="background: #eeeeee; font-family: "courier new" , "courier" , monospace;">protoc --python_out=. addressbook.proto</span></div>
<div>
<br /></div>
<div>
Now used a python script to include the compiled python protobuf header and generated a binary protobuf file called <i>tester_pb</i>. The data contained in it is shown below.<br />
<br />
<span style="font-size: large;">Actual data</span></div>
<div>
<div style="background: #d1eeee;">
[<br />
name: "John Doe"<br />
id: 1234<br />
email: "jdoe@example.com"<br />
id64: -22<br />
uid64: 13360317589766481554<br />
double: 4.5566<br />
bytes: b'\x00\x124V'<br />
]</div>
</div>
<div>
<br /></div>
<div>
<span style="font-size: large;">Protoc output</span> ( <span style="font-family: "courier new" , "courier" , monospace;">protoc --decode_raw < ..\tester_pb</span> )</div>
<div style="background: #eeeeee;">
1 {<br />
1: "John Doe"<br />
2: 1234<br />
3: "jdoe@example.com"<br />
4: <span style="background-color: #ea9999;">18446744073709551594</span><br />
5: 13360317589766481554<br />
6: <span style="background-color: #ea9999;">0x401239f559b3d07d</span><br />
7: <span style="background-color: #ea9999;">"\000\0224V"</span><br />
}</div>
<div>
<br /></div>
<div>
<span style="font-size: large;">protobuf-decoder output</span></div>
<div style="background: #eeeeee;">
{<br />
'01:00:string': 'John Doe',<br />
'02:01:Varint': 1234,<br />
'03:02:string': 'jdoe@example.com',<br />
'04:03:Varint': <span style="background-color: #ea9999;">18446744073709551594</span>,<br />
'05:04:Varint': 13360317589766481554,<br />
'06:05:64-bit': 4.5566<br />
'07:06:<span style="background-color: white;">string</span>': <span style="background-color: #ea9999;">'\x00\x124V'</span><br />
}</div>
<div>
<div>
<br /></div>
</div>
<div>
<span style="font-size: large;">blackboxprotobuf output</span> (includes data dictionary and types dictionary)</div>
<div>
<div style="background: #eeeeee;">
{'1':<br />
{<br />
'1': <span style="background-color: #ea9999;">b</span>'John Doe',<br />
'2': 1234,<br />
'3': <span style="background-color: #ea9999;">b</span>'jdoe@example.com',<br />
'4': -22,<br />
'5': <span style="background-color: #ea9999;">-5086426483943070062</span>,<br />
'6': <span style="background-color: #ea9999;">4616816293942907005</span>,<br />
'7': b'\x00\x124V'<br />
}<br />
}</div>
<div style="background: #eeeeee;">
{'1': {'type': 'message', 'message_typedef':<br />
{<br />
'1': {'type': 'bytes', 'name': ''},<br />
'2': {'type': 'int', 'name': ''},<br />
'3': {'type': 'bytes', 'name': ''},<br />
'4': {'type': 'int', 'name': ''},<br />
'5': {'type': 'int', 'name': ''},<br />
'6': {'type': 'fixed64', 'name': ''},<br />
'7': {'type': 'bytes', 'name': ''}<br />
}, 'name': ''}<br />
}</div>
</div>
<div>
<br /></div>
<div>
As seen in the outputs above, each decoder makes some default assumptions about the data types encountered. The items <span style="background-color: #ea9999;">highlighted in red</span> are the ones that are interpreted using incorrect types. I like blackboxprotobuf because it lets you specify the real type via a <i>types </i>dictionary similar to the one it outputs. So once we have figured out the correct types, we can pass this into the <span style="font-family: "courier new" , "courier" , monospace;">decode_message() </span>function to get the correct output. See code snippet below.<br />
<br /></div>
<div>
<div style="background: #feefe3; line-height: 19px;">
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">import blackboxprotobuf</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">with open('tester_pb', 'rb') as f:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> pb = f.read()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> types = {'1': {'type': 'message', 'message_typedef': <br /> {<br /> '1': {'type': 'str', 'name': 'name'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">'2': {'type': 'int', 'name': 'id'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">'3': {'type': 'str', 'name': 'email'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">'4': {'type': 'int', 'name': 'id64'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">'5': {'type': 'uint', 'name': 'uid64'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">'6': {'type': 'double', 'name': 'double'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">'7': {'type': 'bytes', 'name': 'bytes'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">}, 'name': ''}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> values, _ = blackboxprotobuf.<b>decode_message</b>(pb, types)<br /> print(values)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div style="line-height: 19px;">
<br /></div>
</div>
<div>
That produces the desired output -</div>
<br />
<div style="background: #eeeeee;">
<br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;">{'1':</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> {</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> 'name': 'John Doe',</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> 'id': 1234,</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> 'email': 'jdoe@example.com',</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> 'id64': -22,</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> 'uid64': 13360317589766481554,</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> 'double': 4.5566,</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> 'bytes': b'\x00\x124V'</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: #666666; font-family: "courier new" , "courier" , monospace;">}</span><br />
<br /></div>
<div>
<br /></div>
<div>
In summary, I recommend using the blackboxprotobuf library, <strike>however note that it is not exactly plug and play. Since it is not on <a href="https://pypi.org/" target="_blank">pypi</a>, you have to use it from code.</strike> Also, to use it with python3, I had to make one small tweak. I also added the '<i>str</i>' type decode as that was not available. Since then I have tested this with numerous protobuf streams and it has not failed me! For my updated version of this library, get it <a href="https://github.com/ydkhatri/blackboxprotobuf" target="_blank">here</a>.<br />
<br />
<span style="font-size: large;"><b>Update (8/2020)</b>: I made more bug fixes and published it to pypi, so you can install it via pip now. </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;"> pip install blackboxprotobuf</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;"><br /></span></div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com2tag:blogger.com,1999:blog-1264611260322778486.post-14816674385677092912020-03-28T21:53:00.002-04:002020-03-28T21:54:39.190-04:00Google Search & Personal Assistant data on android<div dir="ltr" style="text-align: left;" trbidi="on">
The Google app, previously known as Google Now, is installed by default on most phones. From the app's description -<br />
<span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;"><br /></span>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBSFPFo-3yczvyNVV9dDuXUZR46A0vTfLvyTj2gs7ndx8PfYZr8E3orGwjdMzwhrCHRjfBJdGhjy82csJKmLxp6-ZtEYvEv9h-e_jXfoJxJ9MpoE2TaiO7S4mrU0PdohMaItA3ZfJ3f-8/s1600/google_app.webp" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="180" data-original-width="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBSFPFo-3yczvyNVV9dDuXUZR46A0vTfLvyTj2gs7ndx8PfYZr8E3orGwjdMzwhrCHRjfBJdGhjy82csJKmLxp6-ZtEYvEv9h-e_jXfoJxJ9MpoE2TaiO7S4mrU0PdohMaItA3ZfJ3f-8/s1600/google_app.webp" /></a><i><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">The Google app keeps you in the know about things that matter to you. Find quick answers, explore your interests, and stay up to date with Discover. The more you use the Google app, the better it gets.</span><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">Search and browse:</span><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">- Nearby shops and restaurants</span><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">- Live sports scores and schedules</span><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">- Movies times, casts, and reviews</span><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">- Videos and images</span><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">- News, stock information, and more</span><br style="-webkit-tap-highlight-color: transparent; background-color: white; color: #333333; font-family: Roboto, Arial, sans-serif; font-size: 14px;" /><span style="background-color: white; color: #333333; font-family: "roboto" , "arial" , sans-serif; font-size: 14px;">- Anything you’d find on the web</span></i><br />
<br />
It is that ubiquitous bar/widget sometimes called the Google Assistant Search Bar or just google Search widget found on the phone's home screen.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5i8APZFJfvW169y6pWxl1Yov-QUh1QsgkDNLogNbqiHYtEsKGQCkZGkHDR5q4NWfmbqolmPyQdqIkh-Wk3aJ2-2tDku-1vf-gJYoY8POyF055EToTAzjKM2ZnnMX90h08ORv0wzfakCw/s1600/searchbar.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="122" data-original-width="421" height="115" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5i8APZFJfvW169y6pWxl1Yov-QUh1QsgkDNLogNbqiHYtEsKGQCkZGkHDR5q4NWfmbqolmPyQdqIkh-Wk3aJ2-2tDku-1vf-gJYoY8POyF055EToTAzjKM2ZnnMX90h08ORv0wzfakCw/s400/searchbar.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Google Search / Personal Assistant Bar </i></td></tr>
</tbody></table>
<br />
The internal package goes by the name com.google.android.googlequicksearchbox. It's artifacts are found at <span style="background-color: yellow; font-family: "courier new" , "courier" , monospace;">/data/data/com.google.android.googlequicksearchbox/</span><br />
<br />
There are many files and folders here, but the most interesting data is the sub-folder <span style="background-color: yellow; font-family: "courier new" , "courier" , monospace;">files/recently</span><br />
<br />
Your recent searches along with some full screen screenshots of search results are stored here. Screenshots (saved as jpg) are in .webp format. The unique number in the name is referenced by the data in the protobuf file (file name is the email address of the logged in user account). If you are not logged in, nothing is populated in this folder. See screenshots below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<span style="text-align: left;"> <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDvyRBV2xQ3HUjWaQ9kIFngTBOhoF0HXyH_CttnVqYD6fY5xsqx-ZMGc-i28U2SXJLjNf5fubL3HKrckXpfGy-TxT8lW08UNmg3-3JIu65hyphenhyphen3GG3l592pFNtDoi8TRYNvIbH0MAc2M0X8/s1600/qsb_no_default_account_highlighted.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="543" data-original-width="376" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDvyRBV2xQ3HUjWaQ9kIFngTBOhoF0HXyH_CttnVqYD6fY5xsqx-ZMGc-i28U2SXJLjNf5fubL3HKrckXpfGy-TxT8lW08UNmg3-3JIu65hyphenhyphen3GG3l592pFNtDoi8TRYNvIbH0MAc2M0X8/s640/qsb_no_default_account_highlighted.png" width="441" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Folder 'recently' has no entries when no account was logged on.</i></td></tr>
</tbody></table>
</span></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzWRqw6CEhdba-IP830YdZQL37Shd-Ev1jTr8-_wbkcQBsUpm-OeE_sYR3tEuHqEvkXXsg5d56FzV-VsfTBNimCq2XfnVDC9iZPmDUW9iKERoKyUCiPme1KhLtKx5x-LJeaJC-rJ8d_CU/s1600/qsb_with_account_2.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="599" data-original-width="387" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzWRqw6CEhdba-IP830YdZQL37Shd-Ev1jTr8-_wbkcQBsUpm-OeE_sYR3tEuHqEvkXXsg5d56FzV-VsfTBNimCq2XfnVDC9iZPmDUW9iKERoKyUCiPme1KhLtKx5x-LJeaJC-rJ8d_CU/s640/qsb_with_account_2.PNG" width="411" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - Folder 'recently' has files when searches were performed after logging in</i></td></tr>
</tbody></table>
<br />
<br />
The protobuf file (<i>jokergogo54@gmail.com</i> in this case) when decoded has entries that look like this (see below) for a typical search. <i>If you aren't familiar with protobuf decoding, read <a href="https://www.swiftforensics.com/2020/01/usagestats-on-android-10-q.html" target="_blank">this</a>.</i><br />
<i><br /></i><span style="color: purple; font-family: "courier new" , "courier" , monospace;">
1 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: 15485946382791341007</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 3: 0</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 4: 1585414188066</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 5: "dolphin"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 8 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: "web"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2: "google.com"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 9: 10449902870035666886</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 17: 1585413397978</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;">}</span><br />
<br />
In the protobuf data (decoded using protoc.exe), as seen above, we can easily distinguish the relevant fields:<br />
<br />
<table style="border-collapse: collapse; border: 1px solid black; padding: 5px;">
<tbody>
<tr>
<th style="border-collapse: collapse; border: 1px solid black; padding: 5px;">Item</th>
<th style="border-collapse: collapse; border: 1px solid black; padding: 5px;">Description</th>
</tr>
<tr>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">1</td>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">session id</td>
</tr>
<tr>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">4</td>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">timestamp1 (unix epoch)</td>
</tr>
<tr>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">5</td>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">search query</td>
</tr>
<tr>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">8</td>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">dictionary<br />
1 = type of search (web, video, ..)<br />
2 = search engine</td>
</tr>
<tr>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">9</td>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">screenshot-id (needs conversion to int from uint)</td>
</tr>
<tr>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">17</td>
<td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">timestamp2 (unix epoch)</td>
</tr>
</tbody></table>
<br />
<br />
Here is the corresponding screenshot saved in the same folder -<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXYoACNSiK7LNnhh-EQThPdpZ7uO8tfgGNpMMhXFqAuCwbfgGbbrmMbpV-2FZmUg6NXo6ABxfKLyJDnES77tF2W2D67RkgGVv9epDuZWyFyDbhzxHwonkY1VGjxZrfSOGV2sYAt8ufP0U/s1600/jokergogo54%2540gmail.com--7996841203673884730.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="901" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXYoACNSiK7LNnhh-EQThPdpZ7uO8tfgGNpMMhXFqAuCwbfgGbbrmMbpV-2FZmUg6NXo6ABxfKLyJDnES77tF2W2D67RkgGVv9epDuZWyFyDbhzxHwonkY1VGjxZrfSOGV2sYAt8ufP0U/s400/jokergogo54%2540gmail.com--7996841203673884730.jpg" width="225" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 4 - Screenshot of search for"dolphin"</i></td></tr>
</tbody></table>
<br />
If you clicked on a recent news story in the app, the protobuf entry looks like this (below):<br />
<br />
<blockquote class="tr_bq">
<span style="color: purple; font-family: "courier new" , "courier" , monospace;">1 {<br /> 1: 9016892896339717414<br /> 3: 1<br /> 4: 1572444614834<br /> 5: ""<br /> 7 {<br /> 1: "https://9to5mac.com/2019/10/30/photos-of-airpods-pro/"<br /> 2: "9to5mac.com"<br /> 3: "Photos of AirPods Pro arriving in stores around the world - 9to5Mac"<br /> }<br /> 9: 9016892896339717414<br /> 10: 9<br /> 17: 1572444614834<br />}</span></blockquote>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD5sjcWgGXisYemmeyiWJ0FaXq2fA6JJS_YSe2r0Yca5xgTDwKnGoB8UZ7zvkZscvJTFVllWGFW3YorEK4sqtRPsDVbkjEPs5rts_52PD45E8q7ye7Od8MNiC6jrHpf4TkRKrL6dm8L58/s1600/xxxxxxx%2540gmail.com-9016892896339717414.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1138" data-original-width="617" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD5sjcWgGXisYemmeyiWJ0FaXq2fA6JJS_YSe2r0Yca5xgTDwKnGoB8UZ7zvkZscvJTFVllWGFW3YorEK4sqtRPsDVbkjEPs5rts_52PD45E8q7ye7Od8MNiC6jrHpf4TkRKrL6dm8L58/s400/xxxxxxx%2540gmail.com-9016892896339717414.jpg" width="216" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 5 - Screenshot for news article clicked from link in google app</i></td></tr>
</tbody></table>
<br />
<br />
Last week, I added a plugin for <a href="https://github.com/abrignoni/ALEAPP" target="_blank">ALEAPP</a> to read these recent search artifacts. This isn't all, there is actually more data to be read here.<br />
<br />
The search widget can be used to make any kind of query, which may then be forwarded to the web browser or Android Auto or the Email or Messaging apps depending on what was queried for. This makes for an interesting artifact.<br />
<br />
From my test data, all searches are stored in the <span style="font-family: "courier new" , "courier" , monospace;"><span style="background-color: yellow;">app_session</span> </span>folder as protobuf files having the extension .binarypb. See screenshot below.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwE_xbkRlMSHbf_JiF6_Mxf5ES0vYC2echQa2NJDhFh-6-Djsl2fUqZE15HLJzkH995K-QEJtjcpnfLocPJJjrvkpimvod5Obbg5wYhaP8ui4Wh0LKj_2bSiv8nVpLhwBX7J9uC8djQGo/s1600/binarypb_files.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="221" data-original-width="279" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwE_xbkRlMSHbf_JiF6_Mxf5ES0vYC2echQa2NJDhFh-6-Djsl2fUqZE15HLJzkH995K-QEJtjcpnfLocPJJjrvkpimvod5Obbg5wYhaP8ui4Wh0LKj_2bSiv8nVpLhwBX7J9uC8djQGo/s1600/binarypb_files.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 6 - .binarypb files</i></td></tr>
</tbody></table>
Each of these files is a protobuf that stores a lot of data about the searches. This includes searches from Android Auto too. Josh Hickman did some excellent research on Android Auto and addressed some of this briefly in his talk <a href="https://dfrws.org/wp-content/uploads/2019/06/2019_USA_pres-android_auto_google_assistant.pdf" target="_blank">here</a>. A parser is not available to read this as the format of the data contained in the protobufs is unknown. I've attempted to reverse-engineer parts of it enough to get the useful bits of information out, such as the search queries. There are also mp3 recordings of the replies from google assistant stored in some of them. These are being added to <a href="https://github.com/abrignoni/ALEAPP" target="_blank">ALEAPP</a> to parse.<br />
<br />
The format here is a bit too much to write about. Below is the raw protobuf structure (sans the binary blobs, replaced by <span style="color: red;">...</span>). The search term here was "tom and jerry".<br />
<br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;">{</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: 0x00000053b0c63c1b</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2: 0x11f0299e</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 3: "search"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 132242267: ""</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 132264001 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: "</span><span style="color: red; font-family: "courier new" , "courier" , monospace;">...</span><span style="color: purple; font-family: "courier new" , "courier" , monospace;">"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2: 0x00000000</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 3: 0</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 4: 0x00000000000e75fe</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 132269388 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2: 0x0000000000000040</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 3 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: "</span><span style="color: red; font-family: "courier new" , "courier" , monospace;">...</span><span style="color: purple; font-family: "courier new" , "courier" , monospace;">"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2: ""</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 3: "and.gsa.launcher.allapps.appssearch"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 132269847 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: "</span><span style="color: red; font-family: "courier new" , "courier" , monospace;">...</span><span style="color: purple; font-family: "courier new" , "courier" , monospace;">"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2: ""</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 3: "and.gsa.launcher.allapps.appssearch"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2 [</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 0: "</span><span style="color: red; font-family: "courier new" , "courier" , monospace;">...</span><span style="color: purple; font-family: "courier new" , "courier" , monospace;">",</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: "</span><span style="color: red; font-family: "courier new" , "courier" , monospace;">...</span><span style="color: purple; font-family: "courier new" , "courier" , monospace;"> tom and jerry </span><span style="color: red; font-family: "courier new" , "courier" , monospace;">...</span><span style="color: purple; font-family: "courier new" , "courier" , monospace;">"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 2: "</span><span style="color: red; font-family: "courier new" , "courier" , monospace;">...</span><span style="color: purple; font-family: "courier new" , "courier" , monospace;">"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 3: 1</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> ]</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 146514374 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: "and.gsa.launcher.allapps.appssearch"</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 206022552 {</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> 1: 0</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> }</span><br />
<span style="color: purple; font-family: "courier new" , "courier" , monospace;">}</span><br />
<br />
After studying this and several other samples, here are the important pieces in the parsed protobuf dictionary:<br />
<br />
<br />
<table style="border-collapse: collapse; border: 1px solid black; padding: 5px;"><tbody>
<tr><th style="border-collapse: collapse; border: 1px solid black; padding: 5px;">Item</th><th style="border-collapse: collapse; border: 1px solid black; padding: 5px;">Description</th></tr>
<tr><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">1</td><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">session id (same number as in filename)</td></tr>
<tr><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">3</td><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">type of query (search, car_assistant, opa)<br />
car_assistant = Android Auto<br />
opa = personal assistant</td></tr>
<tr><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">132269388</td><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">dictionary<br />
1 = mp3 recording of response</td></tr>
<tr><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">132269847</td><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">dictionary<br />
<br />
1 = dictionary<br />
<table style="border-collapse: collapse; border: 1px solid black; padding: 5px;"><tbody>
<tr><th style="border-collapse: collapse; border: 1px solid black; padding: 5px;">Item</th><th style="border-collapse: collapse; border: 1px solid black; padding: 5px;">Description</th></tr>
<tr><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">2</td><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;">last query</td></tr>
</tbody></table>
<br />
2 = List of session queries (in blobs)</td></tr>
<tr><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;"><br /></td><td style="border-collapse: collapse; border: 1px solid black; padding: 5px;"></td></tr>
</tbody></table>
<br />
<br />
For more details, refer the module googleQuickSearchbox.py in <a href="https://github.com/abrignoni/ALEAPP" target="_blank">ALEAPP</a>. Below is a screenshot of the parsed out data.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHYOyeYP6hD9on0uiGhSLu9KayeFSg-SZnPi5ouYWTZ8wzF-k1BQ2Y47uk_YIWcox5qITxCSNyCK-a0P1hidvMzVBvWuXkKv3Mot3lVrtM1djZeK3hzSI4B4Tv4ZXj5zFmD0ZZZSQfBKk/s1600/aleapp.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="637" data-original-width="1573" height="257" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgHYOyeYP6hD9on0uiGhSLu9KayeFSg-SZnPi5ouYWTZ8wzF-k1BQ2Y47uk_YIWcox5qITxCSNyCK-a0P1hidvMzVBvWuXkKv3Mot3lVrtM1djZeK3hzSI4B4Tv4ZXj5zFmD0ZZZSQfBKk/s640/aleapp.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 7 - ALEAPP output showing Google App / Personal assistant queries</i></td></tr>
</tbody></table>
<br /></div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-33981514387431593612020-01-08T00:00:00.000-05:002020-01-08T13:28:12.650-05:00Usagestats on Android 10 (Q)<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
UsageStats</h2>
If you are unfamiliar with this artifact, Alex Brignoni explains the UserStats artifact in the blog post <a href="https://abrignoni.blogspot.com/2019/02/android-usagestats-xml-parser.html" target="_blank">here</a>. Located at <b>/data/system/usagestats/</b> this information can be useful in cases. Up until Android 9 (Pie), this was in XML format, however since Android 10(Q), it is now in a different format. So the tool written by Alex didn't work out for me or my students investigating this artifact a couple of months back.<br />
<br />
The file name has the same format (unix millisecond time as integer) and below you can see what the new data looks like.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCjCuAEZdmb2Ftm9gLElT2xHIdMZuRD6vxv2xx6Ahl_4VzFjg05mWvk6pnUQ6tx1J3G1UPfOdoRWpVR_VCPhzNzDkLSvRm_5GHNK8wcWwkQC_SbCg4wBY-7r12hICKUg4r_7_bs5HeI90/s1600/usagestats-raw_data.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="129" data-original-width="594" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCjCuAEZdmb2Ftm9gLElT2xHIdMZuRD6vxv2xx6Ahl_4VzFjg05mWvk6pnUQ6tx1J3G1UPfOdoRWpVR_VCPhzNzDkLSvRm_5GHNK8wcWwkQC_SbCg4wBY-7r12hICKUg4r_7_bs5HeI90/s1600/usagestats-raw_data.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - File 1572840777639 - raw hex view (complete file not shown)</i></td></tr>
</tbody></table>
It appeared to be some sort of binary format, but without a standard consistent header (after I compared a few files). Taking a cue from fellow DFIR researchers (<a href="https://www.mac4n6.com/blog/" target="_blank">Sarah Edwards</a> and <a href="https://thinkdfir.com/" target="_blank">Phill Moore</a>), I tested if this was a protocol buffer. If you aren't familiar with a Protocol Buffer, read these posts from <a href="https://www.mac4n6.com/blog/2019/9/27/just-call-me-buffy-the-proto-slayer-an-initial-look-into-protobuf-data-in-mac-and-ios-forensics" target="_blank">Sarah</a> and <a href="https://thinkdfir.com/2019/01/11/what-did-i-listen-to-on-spotify-for-ios/" target="_blank">Phill</a>. This is a <a href="https://developers.google.com/protocol-buffers" target="_blank">google</a> creation, and as they describe it - .<i>..a language-neutral, platform-neutral extensible mechanism for serializing structured data.</i><br />
<br />
To test for protocol buffer presence (on windows), you will need to download protoc.exe from <a href="https://github.com/protocolbuffers/protobuf/releases" target="_blank">here</a>. Run protoc.exe as shown below. Here 1572840777639 is the filename. If you got output, its a protobuf.<br />
<br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">W:\usagestats\0\daily><b>protoc --decode_raw < 1572840777639</b></span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">1: 1862148</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">3: 1</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">4: 1</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">2 {</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 1: 74</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 2: "com.google.android.youtube"</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 2: "com.google.android.ext.services"</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 2: "com.android.providers.telephony"</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 2: "com.android.dynsystem"</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 2: "com.android.settings.CryptKeeper"</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">...</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">...output snipped...</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">...</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">22 {</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 2: 23</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 4: 60</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 5: 1249881</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 7: 23</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 14: 92830887</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 15: 23</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;"> 16: 60</span><br />
<span style="color: #444444; font-family: "courier new" , "courier" , monospace;">}</span><br />
<br />
OK, so we got some decoded json data back. But it still did not look like anything we are used to seeing (see XML below).<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg86mhyweC_1GehGzAb8SG-Vun9L226Qn-QKwwTE3OiKg0AkzY3fCBEk1FWoJAqECGnC9-OerNLSqzrN_rG_OJP0EHBww1e3HuA8S4rH7B6QuSLi0v9HMYVlel71aX74cdPa2AtW4wyRig/s1600/usagestats-xml.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="179" data-original-width="961" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg86mhyweC_1GehGzAb8SG-Vun9L226Qn-QKwwTE3OiKg0AkzY3fCBEk1FWoJAqECGnC9-OerNLSqzrN_rG_OJP0EHBww1e3HuA8S4rH7B6QuSLi0v9HMYVlel71aX74cdPa2AtW4wyRig/s1600/usagestats-xml.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - XML usagestats snippet</i></td></tr>
</tbody></table>
The way protocol buffers work, you need a .proto file that defines the structure and data types of the data contained in the buffer. So to decode this, we need the .proto file!<br />
<br />
Since Android is open source, so why not peek at the source code of AOSP? To avoid downloading the entire source code, just browse the aosp-mirror on github.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvVfx-frP77slqSx3m0YtKGEtfbQuPdO59lNGwb3Gs_hgVZ9JgWsFd-TpS89wuQin04knCUVV8v_O9QYeq9YM7uyCgz6w6MAN_b894mbG3tuCp_EF9U4fE_N3uRRr4fT7vTDflNoihtGo/s1600/aosp-framework-github.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="354" data-original-width="709" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvVfx-frP77slqSx3m0YtKGEtfbQuPdO59lNGwb3Gs_hgVZ9JgWsFd-TpS89wuQin04knCUVV8v_O9QYeq9YM7uyCgz6w6MAN_b894mbG3tuCp_EF9U4fE_N3uRRr4fT7vTDflNoihtGo/s1600/aosp-framework-github.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - aosp source code on github</i></td></tr>
</tbody></table>
<br />
After a bit of searching, we find the file we are looking for at:<br />
<b>platform_frameworks_base/core/proto/android/server/usagestatsservice.proto</b><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjP7fQRwLPanmfWHMFAByXVF_2hRs_IiqvLPiyP6nLiSlo_LmXv2XR_53KoQhWbAwDofTuoRsPwAno06DLs3Ph0LDibJ_E__obF6_z3lGTPm2d4WsDLPeAJczZ6DBgp-NncfB2Z-cXlJ9A/s1600/usagestatsproto.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="685" data-original-width="643" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjP7fQRwLPanmfWHMFAByXVF_2hRs_IiqvLPiyP6nLiSlo_LmXv2XR_53KoQhWbAwDofTuoRsPwAno06DLs3Ph0LDibJ_E__obF6_z3lGTPm2d4WsDLPeAJczZ6DBgp-NncfB2Z-cXlJ9A/s640/usagestatsproto.PNG" width="600" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 4 - usagestatsservice.proto file snippet</i></td></tr>
</tbody></table>
As seen above, the file references other .proto files too. So we must get those too, and any dependencies in those as well (recursively). We eventually end up with 7 files:<br />
<br />
<ul style="text-align: left;">
<li>usagestatsservice.proto</li>
<li>configuration.proto</li>
<li>privacy.proto</li>
<li>locale.proto</li>
<li>rect.proto</li>
<li>protobuf_descriptor.proto</li>
<li>window_configuration.proto</li>
</ul>
<br />
Next, we need to transform (google says <a href="https://developers.google.com/protocol-buffers/docs/pythontutorial#compiling-your-protocol-buffers" target="_blank">compile</a>) our .proto files into python libraries. Use protoc.exe to do so. The syntax is :<br />
<br />
<pre dir="ltr" is-upgraded="" style="background: rgb(241, 243, 244); box-sizing: inherit; color: #37474f; font-family: "Roboto Mono", monospace; font-size: 14px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: 20px; overflow-x: auto; padding: 24px; position: relative;" translate="no"><b>protoc -I=</b>$SRC_DIR <b>--python_out=</b>$DST_DIR $SRC_DIR/<i>your_proto_file</i>.proto</pre>
<br />
Do this for every .proto file. It will generate a .py file for each one. For example, the usagestatsservice.proto compiles to usagestatsservice_pb2.py. Now all that remains is to use these generated python files to read our raw protocol buffer from file. We will need to write some code to do so.<br />
<br />
Peeking into the usagestatsservice.proto file, you get some idea of how this might work. I constructed a basic python script to read this (below).
<br />
<div style="background-color: white; font-family: consolas, "courier new", monospace; font-size: 14px; white-space: pre;">
<div style="font-family: Consolas, "Courier New", monospace; line-height: 19px;">
<span style="color: #af00db;">import</span> usagestatsservice_pb2<br />
input_path = <span style="color: #a31515;">"W:</span><span style="color: red;">\\</span><span style="color: #a31515;">usagestats</span><span style="color: red;">\\</span><span style="color: #a31515;">0</span><span style="color: red;">\\</span><span style="color: #a31515;">daily</span><span style="color: red;">\\</span><span style="color: #a31515;">1572840777639"</span>
stats = usagestatsservice_pb2.IntervalStatsProto()
<span style="color: #af00db;">with</span> <span style="color: #795e26;">open</span> (input_path, <span style="color: #a31515;">'rb'</span>) <span style="color: #af00db;">as</span> f:
stats.ParseFromString(f.read())
<span style="color: green;"># GET PACKAGES</span>
<span style="color: #af00db;">for</span> usagestat <span style="color: blue;">in</span> stats.packages:
<span style="color: #795e26;">print</span>(<span style="color: #a31515;">'package = '+ </span>stats.stringpool.strings[usagestat.package_index - <span style="color: #09885a;">1</span>])
<span style="color: #795e26;">print</span>(usagestat)<br />
<span style="color: green;"> # GET CONFIGURATIONS</span>
<span style="color: #af00db; font-family: "consolas" , "courier new" , monospace;">for</span><span style="font-family: "consolas" , "courier new" , monospace;"> conf </span><span style="color: blue; font-family: "consolas" , "courier new" , monospace;">in</span><span style="font-family: "consolas" , "courier new" , monospace;"> stats.configurations:</span>
<span style="color: #795e26;">print</span>(<span style="font-family: "consolas" , "courier new" , monospace;">conf</span>)<br />
<span style="color: green;"> # GET EVENT LOGS</span>
<span style="color: #af00db; font-family: "consolas" , "courier new" , monospace;">for</span><span style="font-family: "consolas" , "courier new" , monospace;"> </span><span style="font-family: "consolas" , "courier new" , monospace;">event </span><span style="color: blue; font-family: "consolas" , "courier new" , monospace;">in</span><span style="font-family: "consolas" , "courier new" , monospace;"> stats.event_log</span><span style="font-family: "consolas" , "courier new" , monospace;">:</span>
<span style="color: #795e26;">print</span>(<span style="font-family: "consolas" , "courier new" , monospace;">event</span>)
</div>
</div>
<div>
<br />
You can check for the existence of a field using the <span style="font-family: "courier new" , "courier" , monospace;">HasField()</span> function. So here is what a package object consists of:<br />
<blockquote class="tr_bq">
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>package = com.android.settings<br />package_index: 58<br />last_time_active_ms: 663647<br />total_time_active_ms: 4897<br />app_launch_count: 3<br />last_time_service_used_ms: -1572840673324<br />last_time_visible_ms: 673237<br />total_time_visible_ms: 25221</i></span></blockquote>
A configuration object consists of:<br />
<blockquote class="tr_bq">
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>config {</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> font_scale: 1.0</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> locales {</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> language: "en"</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> country: "US"</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> }</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> screen_layout: 268435794</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> color_mode: 5</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> touchscreen: 3</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> keyboard: 2</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> keyboard_hidden: 1</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> hard_keyboard_hidden: 1</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> navigation: 1</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> navigation_hidden: 2</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> orientation: 1</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> screen_width_dp: 411</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> screen_height_dp: 659</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> smallest_screen_width_dp: 411</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> density_dpi: 560</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> window_configuration {</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> app_bounds {</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> right: 1440</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> bottom: 2392</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> }</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> windowing_mode: 1</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> bounds {</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> right: 1440</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> bottom: 2560</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> }</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i> }</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>}</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>last_time_active_ms: 662163</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>total_time_active_ms: 37</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>count: 1</i></span></blockquote>
<br />
An event log object contains:<br />
<blockquote class="tr_bq">
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>package = com.google.android.apps.nexuslauncher</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>class = com.google.android.apps.nexuslauncher.NexusLauncherActivity</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>task root package = com.google.android.apps.nexuslauncher</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>task root class = com.google.android.apps.nexuslauncher.NexusLauncherActivity</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>type = MOVE_TO_FOREGROUND</i></span><br />
<span style="color: blue; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i>time_ms: 34440</i></span></blockquote>
So now, our protobuf parsed and file read and interpreted successfully! That's it for now. <i>On to the next artifact.. </i></div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-84519463614311733902019-10-28T22:31:00.001-04:002019-10-28T22:34:38.319-04:00macOS 10.15 Volumes & Firmlink magic<div dir="ltr" style="text-align: left;" trbidi="on">
With macOS 10.15 - Catalina, Apple has introduced a change in the way system and user data is stored on disk. In prior versions, the root '<b>/</b>' volume was stored in a single volume usually named '<b>Macintosh HD</b>'. This did not change with the update to APFS. However with Catalina, there are now two distinct volumes -<br />
<ul style="text-align: left;">
<li>Macintosh HD</li>
<li>Macintosh HD - Data</li>
</ul>
The screenshot below shows the two different volumes -<br />
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjasJdkIgfo2mHThRuBgdEo6LOoeK7gWSh3oCA-D9-VCOPoq9T44KHaiU_2g9nByglMTkIivWxNDRCyK8JSHkqa_JjBlEXgdXoMQqATmhZGJhEJoisDa34n48VDMsE-FrrxhTLLntUnoXQ/s1600/Screen+Shot+2019-10-28+at+9.37.46+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="670" data-original-width="1312" height="324" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjasJdkIgfo2mHThRuBgdEo6LOoeK7gWSh3oCA-D9-VCOPoq9T44KHaiU_2g9nByglMTkIivWxNDRCyK8JSHkqa_JjBlEXgdXoMQqATmhZGJhEJoisDa34n48VDMsE-FrrxhTLLntUnoXQ/s640/Screen+Shot+2019-10-28+at+9.37.46+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - diskutil output showing a split Macintosh HD volume into two</i></td></tr>
</tbody></table>
<br />
The <b>Macintosh HD</b> volume stores the system files and is mounted as read-only, while the <b>Macintosh HD - Data</b> volume has all the other files on your system which include user profiles, system and user data, and user installed Applications.<br />
<br />
However when booted, only a single logical volume is presented (as root /) that combines the contents of both. This is enabled through APFS using its Volume Role feature. This is mentioned in <a href="https://developer.apple.com/support/downloads/Apple-File-System-Reference.pdf" target="_blank">Apple's official APFS documentation</a> but its usage or working is not documented. Each volume can be assigned a role in its volume Superblock structure (apfs_superblock_t). There are 8 possible roles documented. From the Apple docs:<br />
<br />
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> #define APFS_VOL_ROLE_NONE 0x0000</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> #define APFS_VOL_ROLE_SYSTEM 0x0001</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;">#define APFS_VOL_ROLE_USER 0x0002</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;">#define APFS_VOL_ROLE_RECOVERY 0x0004</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;">#define APFS_VOL_ROLE_VM 0x0008</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;">#define APFS_VOL_ROLE_PREBOOT 0x0010</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;">#define APFS_VOL_ROLE_INSTALLER 0x0020</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;">#define APFS_VOL_ROLE_DATA 0x0040</span></div>
<div style="text-align: left;">
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;"> </span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace;">#define APFS_VOL_ROLE_BASEBAND 0x0080</span></div>
<br />
The SYSTEM volume contains the folders <b>/bin</b>, <b>/sbin</b> and most of the <b>/usr</b> and <b>/System</b> folders. A few subfolders of <b>/usr</b> and <b>/System</b> are on the DATA volume. The volumes are joined using a new construct that Apple calls firmlinks. They describe it as a <i>Bi-directional wormhole in path traversal. Firmlinks are used on the system volume to point to the user data on the data volume.</i><br />
<br />
They are somewhat similar to the unix symlinks and hardlinks, but only directories can be linked (from one volume to another). The file that defines/lists the firmlinks resides on the SYSTEM volume at <b>/usr/share/firmlinks</b>. The following paths are defined by default.<br />
<br />
<div class="p1">
<span class="s1"><span style="color: #666666;"> /AppleInternal<span class="Apple-tab-span"> </span>AppleInternal</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /Applications<span class="Apple-tab-span"> </span>Applications</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /Library<span class="Apple-tab-span"> </span>Library</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /System/Library/Caches<span class="Apple-tab-span"> </span>System/Library/Caches</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /System/Library/Assets<span class="Apple-tab-span"> </span>System/Library/Assets</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /System/Library/PreinstalledAssets<span class="Apple-tab-span"> </span>System/Library/PreinstalledAssets</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /System/Library/AssetsV2<span class="Apple-tab-span"> </span>System/Library/AssetsV2</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /System/Library/PreinstalledAssetsV2<span class="Apple-tab-span"> </span>System/Library/PreinstalledAssetsV2</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /System/Library/CoreServices/CoreTypes.bundle/Contents/Library </span></span><br />
<span class="s1"><span style="color: #666666;"> System/Library/CoreServices/CoreTypes.bundle/Contents/Library</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /System/Library/Speech<span class="Apple-tab-span"> </span>System/Library/Speech</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /Users<span class="Apple-tab-span"> </span>Users</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /Volumes<span class="Apple-tab-span"> </span>Volumes</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /cores<span class="Apple-tab-span"> </span>cores</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /opt<span class="Apple-tab-span"> </span>opt</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /private<span class="Apple-tab-span"> </span>private</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /usr/local<span class="Apple-tab-span"> </span>usr/local</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /usr/libexec/cups<span class="Apple-tab-span"> </span>usr/libexec/cups</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #666666;"> /usr/share/snmp<span class="Apple-tab-span"> </span>usr/share/snmp</span></span></div>
<br />
The linked volumes will have distinct inode numbers for files/folders. The only common inode numbers seen are inodes 1 (<i>Parent of root</i>), 2 (<i>root</i>) and 3 (<i>private-dir</i>). All other inodes will be unique, a simple but clever scheme is used to ensure that. For the SYSTEM volume, every inode number allocated will be OR'd with 0x0FFFFFFF00000000. Take a look at the inode numbers in the combined volume in screenshot below. The very large numbers are the files that reside on the SYSTEM volume due to the upper bits being set by the mask.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiHLDyHB_3qQ8z62ytPZ1hV1jympcHpuD4Co0pE8EGvKFy8RvFhOGgZBlSjfMw8mbOeCRQot612Na4HgrlfIjdu14o0q_UoK_BmxXGWkXrbrJolLeXjYSub4pfOvIkw586IsqsaPnjET8/s1600/Screen+Shot+2019-10-28+at+9.17.15+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="570" data-original-width="1340" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiHLDyHB_3qQ8z62ytPZ1hV1jympcHpuD4Co0pE8EGvKFy8RvFhOGgZBlSjfMw8mbOeCRQot612Na4HgrlfIjdu14o0q_UoK_BmxXGWkXrbrJolLeXjYSub4pfOvIkw586IsqsaPnjET8/s640/Screen+Shot+2019-10-28+at+9.17.15+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Contents of root showing files from both SYSTEM and DATA</i></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
If you try to create a file or folder on the root volume (or one of its owned folders), it fails with an error: <i>Read-only volume.</i><br />
<i><br /></i>
For accessing most files and folders, there should be no problem as the stitched/combined volume works seamlessly so all programs should not notice any difference. However there are situations where you might want to explicitly access a folder from a specific volume, especially for forensics. For example, if you wanted to access <b>/./fseventsd</b>, you would always get the read-only volume's .<b>fseventsd</b> folder which won't be too interesting as its a read-only volume! To get the one on the DATA volume, there is still a way. Apple has also made the DATA volume available (mounted) at the mountpoint <b>/System/Volumes/Data</b>. <u>This also means that if you have scripts that run across all files, they will need to be made aware that this location should be avoided to prevent duplication.</u><br />
<br />
According to <a href="https://devstreaming-cdn.apple.com/videos/wwdc/2019/710aunvynji5emrl/710/710_whats_new_in_apple_file_systems.pdf" target="_blank">Apple</a>, you cannot opt-out of this, and it is a required feature for macOS 10.15. Forensic tools that operate on full disk images will have to adapt for this change, and so I've updated <a href="https://github.com/ydkhatri/mac_apt" target="_blank">mac_apt</a> to support macOS Catalina. If you use it, let me know of bugs/issues.<br />
<br /></div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com2tag:blogger.com,1999:blog-1264611260322778486.post-58456903402952875452019-10-26T20:01:00.001-04:002019-10-26T22:41:41.810-04:00Part 3 - ADB keyvalue backups - Wifi and System settings <div dir="ltr" style="text-align: left;" trbidi="on">
<div>
This is Part 3 of the continuing blog series on <a href="https://www.swiftforensics.com/2019/10/adb-keyvalue-backups-and-data-format.html" target="_blank">ADB keyvalue backups</a>. Today we focus on Wifi settings and other system configuration available from-</div>
<blockquote class="tr_bq">
<span style="background-color: yellow;"><span style="font-family: inherit;"><i><Backup.adb></i><b>/apps/com.android.providers.settings/k/com.android.providers.settings.data</b></span></span></blockquote>
<div>
<i>They are backed up only if you specified the -keyvalue option <u>and are available on non-rooted devices too</u>.</i></div>
<div>
<i><br /></i></div>
<div>
This file follows the Key-Value Data format as outlined earlier in <a href="https://www.swiftforensics.com/2019/10/adb-keyvalue-backups-and-data-format.html" target="_blank">part 1</a>. There are 8 different types of data seen here. The Key name represents the type of data and Value represents either a single structure or a set of name-value pairs (both name and value are strings). The table below shows the data seen here.<br />
<br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #F4B083; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Key Name<o:p></o:p></div>
</td>
<td style="background: #F4B083; border-left: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Description<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>system<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
settings about font sizes, screen brightness, hearing aids, haptic
feedback among others<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>secure<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
more system settings on gestures, button behaviors, spell checker,
screensaver, accessibility, etc..<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>global<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Boolean settings that enable/disable options like wifi wakeup, auto_time, sounds enabled, call
auto-retry, etc..<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>locale<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
a locale string like ‘en-US’<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>lock_settings<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
owner info for display on screen if enabled<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>softap_config<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Access point settings for Mobile hotspot<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>network_policies<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
unknown<o:p></o:p></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 116.75pt;" valign="top" width="156"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<b>wifi_new_config<o:p></o:p></b></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 333.0pt;" valign="top" width="444"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
xml data having wifi settings for connected access points<o:p></o:p></div>
</td>
</tr>
</tbody></table>
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Perhaps the most interesting aspect here is the presence of wifi passwords (WPA pre-shared keys) in the <span style="font-family: inherit;"><b>wifi_new_config </b></span>data as see in screenshot below. <i>Yes, you can get wifi passwords from an adb backup now!</i></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img border="0" data-original-height="505" data-original-width="624" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpjmy5NfABjTcyfBuxz6ILN2uFvqllmhGJe9HO6TUuVCc7r7wA0sLOVUIpOrDa2JYslUR8fBYFyVsk9jkSnX1NU5Odhb1KvqPsSLVVUX6eVXqPelWELfSUbFxOQPpX9E9ix0freFITf08/s1600/wifi_info_android_backups.png" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Snippet of Wifi saved settings from com.android.providers.settings.data showing SSIDs & passwords</i></td></tr>
</tbody></table>
<a href="https://github.com/ydkhatri/Android/blob/master/adb_backup_parser/providers_settings_parser.py" target="_blank">Here is a python script</a> to read <b>com.android.providers.settings.data</b> and export the information to json files. Below you can see some of the data parsed by the this script for one of my test devices.<br />
<div>
<br />
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYiu-p1rg26ibdEE0CyhkLk8LTe7LUcy4YYGyI2WHWu3lYcBMVMdIc7oSzczvIK5nBR3SOEVB_CQFt2-rwCkskpTV44bz0cDyLFFksRhSOJAdMOc0driKhD8gB_Z_HIt7RTAnh46N1Ksk/s1600/global.PNG" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="332" data-original-width="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYiu-p1rg26ibdEE0CyhkLk8LTe7LUcy4YYGyI2WHWu3lYcBMVMdIc7oSzczvIK5nBR3SOEVB_CQFt2-rwCkskpTV44bz0cDyLFFksRhSOJAdMOc0driKhD8gB_Z_HIt7RTAnh46N1Ksk/s1600/global.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Data from 'global' key</i></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh13OGBXllf8xgAXJ2DrdwgcPJIgn2BkABbkK3fYNhsEPU6h0WhD5nfUmtH4SKqCNMl94PyUXo2cvF4hn6rrQMzpBeats_2mAL4CsJyENeo9gE5iPuP3TspvZUiLv-CnWt0XnER4VnkoqA/s1600/system.PNG" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="370" data-original-width="582" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh13OGBXllf8xgAXJ2DrdwgcPJIgn2BkABbkK3fYNhsEPU6h0WhD5nfUmtH4SKqCNMl94PyUXo2cvF4hn6rrQMzpBeats_2mAL4CsJyENeo9gE5iPuP3TspvZUiLv-CnWt0XnER4VnkoqA/s1600/system.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i style="font-size: 12.8px;">Figure 3 - Data from 'system' key</i></td></tr>
</tbody></table>
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKcDz7ndIlgQUeOa7nodE-eZr43ZdrvgZJ9XLTQvSe0bZRAL-I9-v0mCsQG45nIMj9BXsxij0R8Do4mEPQ-II5K-Udsfwdj64Njy5zyVTe6dlFPRQgBsztJZDdY_EZ7SKuP228QeUZCsw/s1600/secure.PNG" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="178" data-original-width="661" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKcDz7ndIlgQUeOa7nodE-eZr43ZdrvgZJ9XLTQvSe0bZRAL-I9-v0mCsQG45nIMj9BXsxij0R8Do4mEPQ-II5K-Udsfwdj64Njy5zyVTe6dlFPRQgBsztJZDdY_EZ7SKuP228QeUZCsw/s1600/secure.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i style="font-size: 12.8px;">Figure 4 - Data from 'secure' key</i></td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZp4vwWn1sPXPbW9iiW-Oys9ChkvP77yFR1u1tAaO9wcorwzw3p0M0tvNOCZdsEeg3FEzmJzd_sBWGreA6goRtuO3uXqfDU8qDbKqbB4VuSfnGGL7JgCCbTdcDtZUxLmpF72e7Lvojq84/s1600/softap_config.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="120" data-original-width="227" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZp4vwWn1sPXPbW9iiW-Oys9ChkvP77yFR1u1tAaO9wcorwzw3p0M0tvNOCZdsEeg3FEzmJzd_sBWGreA6goRtuO3uXqfDU8qDbKqbB4VuSfnGGL7JgCCbTdcDtZUxLmpF72e7Lvojq84/s1600/softap_config.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i style="font-size: 12.8px;">Figure 5 - Data from 'softap_config' key</i></td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjp0M1M0ql0jUymVSC9flOS7UlomX_Q6kH5tn670NcbWs_6YI-FQfMZa68z32nxvonE1XRGOCaSx3OnYyBY0eHU7RNjTysBn8ajCa_Jsh52NoppPnUvDoui5ZgE-8yQjQCMO6boGIXH2HE/s1600/lock_settings.PNG" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" data-original-height="90" data-original-width="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjp0M1M0ql0jUymVSC9flOS7UlomX_Q6kH5tn670NcbWs_6YI-FQfMZa68z32nxvonE1XRGOCaSx3OnYyBY0eHU7RNjTysBn8ajCa_Jsh52NoppPnUvDoui5ZgE-8yQjQCMO6boGIXH2HE/s1600/lock_settings.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i style="font-size: 12.8px;">Figure 6 - Data from 'lock_settings' key</i></td></tr>
</tbody></table>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com1tag:blogger.com,1999:blog-1264611260322778486.post-47748534724813893772019-10-15T01:31:00.000-04:002019-10-15T01:31:45.383-04:00Part 2 - ADB keyvalue backups - Call Logs<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
This is Part 2 of the continuing blog series on <a href="https://www.swiftforensics.com/2019/10/adb-keyvalue-backups-and-data-format.html" target="_blank">ADB keyvalue backups</a>. Today we focus on Call Log Backups. </div>
<div>
<br /></div>
<div>
Call logs are backed up under </div>
<blockquote class="tr_bq">
<span style="background-color: yellow;"><span style="font-family: inherit;"><i><Backup.adb></i><b>/apps/com.android.calllogbackup/k/com.android.calllogbackup.data</b></span></span></blockquote>
<div>
<i>They are backed up only if you specified the -keyvalue option <u>and are available on non-rooted devices too</u>.</i></div>
<div>
<i><br /></i></div>
<div>
This file follows the Key-Value Data format as outlined earlier in <a href="https://www.swiftforensics.com/2019/10/adb-keyvalue-backups-and-data-format.html" target="_blank">part 1</a>. The Keys here are the call ids or serial number of calls, starting at 1 and sequentially rising. The Values are the individual call log records.</div>
<div>
<br /></div>
<div>
Here are the structures used in the Call Log record. All fields here are stored as Big Endian.</div>
<div>
<br /></div>
<div>
<b>1. Text_Record</b></div>
<div>
<br /></div>
<div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td style="background: #F4B083; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Position<o:p></o:p></div>
</td>
<td style="background: #F4B083; border-left: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 90.65pt;" valign="top" width="121">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Type<o:p></o:p></div>
</td>
<td style="background: #F4B083; border-left: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 152.35pt;" valign="top" width="203">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Description<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 1;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
00<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 90.65pt;" valign="top" width="121">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
ushort<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 152.35pt;" valign="top" width="203">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
field_length (in bytes)<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 2; mso-yfti-lastrow: yes;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
02<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 90.65pt;" valign="top" width="121">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
char[field_length]<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 152.35pt;" valign="top" width="203">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
field data (text)<o:p></o:p></div>
</td>
</tr>
</tbody></table>
</div>
<div>
<br /></div>
<div>
<b>2. Call_Log</b></div>
<div>
<br /></div>
<div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="mso-yfti-firstrow: yes; mso-yfti-irow: 0;">
<td style="background: #F4B083; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Position<o:p></o:p></div>
</td>
<td style="background: #F4B083; border-left: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Type<o:p></o:p></div>
</td>
<td style="background: #F4B083; border-left: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Description<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 1;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
00<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
uint<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
version, 0x03EF (1007) or 1005 seen<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 2;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
04<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
int64<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
timestamp<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 3;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
12<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
uint64<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
call duration in seconds<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 4;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
20<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
byte<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
is_phone_number_present<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 5;">
<td style="background: #FFF2CC; border-top: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
21<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Text_Record<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
present if is_phone_number_present = 1<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 6;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
uint<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
call type<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>1 = Incoming<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>2 = Outgoing<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>3 = Missed<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>4 = voicemail<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>5 = Rejected / Declined<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>6 = Blocked<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>7 = Answered_Externally<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 7;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
uint<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
number presentation<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>1 = Allowed<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>2 = Restricted<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>3 = Unknown<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>4 = Payphone<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 8;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
byte<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
is_servicename_present<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 9;">
<td style="background: #FFF2CC; border-top: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Text_Record<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
present if is_ servicename_present = 1<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 10;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
byte<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
is_iccid_present<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 11;">
<td style="background: #FFF2CC; border-top: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Text_Record<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
present if is_ iccid _present = 1<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 12;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
byte<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
is_own_num_present<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 13;">
<td style="background: #FFF2CC; border-top: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Text_Record<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
present if is_ iccid _present = 1<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 14;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
byte[12]<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
unknown bytes, always 0<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 15;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
Text_Record<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
oem namespace string<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 16;">
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
byte[18]<o:p></o:p></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
unknown bytes<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 17;">
<td style="background: #FFF2CC; border-top: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
uint<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
block reason (only on version 1007)<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>1 = Screening service<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>2 = Direct to voicemail<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>3 = Blocked number<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>4 = Unknown number<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>5 = Restricted number<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>6 = Payphone<o:p></o:p></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="mso-spacerun: yes;"> </span>7 = Not in contacts<o:p></o:p></div>
</td>
</tr>
<tr style="mso-yfti-irow: 18; mso-yfti-lastrow: yes;">
<td style="background: #FFF2CC; border-top: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
..<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="100">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
byte[18]<o:p></o:p></div>
</td>
<td style="background: #FFF2CC; border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-background-themecolor: accent4; mso-background-themetint: 51; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 217.8pt;" valign="top" width="290">
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
unknown bytes (only on version 1007)<o:p></o:p></div>
</td>
</tr>
</tbody></table>
</div>
<div>
<br /></div>
<div>
The screenshot below shows a raw record in the hex editor.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaDbVnun314PJ1bZP8Nja406sUdPRoB7eotxGqJQv4VWjRPbnJqJBe4YmfnMcJNagVwfjdV77PMTNeslgRG9s7uuM3Gc9I4wcEypZDWX4sar1eFjbJD2l_p_rA4Tymdo3XbkBhu9vpl4Y/s1600/calllog.data.hex.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="414" data-original-width="756" height="350" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaDbVnun314PJ1bZP8Nja406sUdPRoB7eotxGqJQv4VWjRPbnJqJBe4YmfnMcJNagVwfjdV77PMTNeslgRG9s7uuM3Gc9I4wcEypZDWX4sar1eFjbJD2l_p_rA4Tymdo3XbkBhu9vpl4Y/s640/calllog.data.hex.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Call log record showing some important fields</i></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Using an 010 template to parse this information, it looks like this (below).</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLX_aQfFt4YbFWuOmiH-GNCKTIf4cjyLidIQ1TFGsi2JCRhD2MNA3gzdu3QB95kQWaRDcisaWxw9Tip3iG-ZDOaDisBXN5CuZYhB-Z9EPPVpowleXf4PhDtoh-3pbUk6_1pv7D2QlwHrw/s1600/calllog.data.parsed.010.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="440" data-original-width="692" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLX_aQfFt4YbFWuOmiH-GNCKTIf4cjyLidIQ1TFGsi2JCRhD2MNA3gzdu3QB95kQWaRDcisaWxw9Tip3iG-ZDOaDisBXN5CuZYhB-Z9EPPVpowleXf4PhDtoh-3pbUk6_1pv7D2QlwHrw/s1600/calllog.data.parsed.010.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i style="font-size: 12.8px;">Figure 2 - Call log record data parsed in 010 editor</i></td></tr>
</tbody></table>
<div>
The level of detail on these records is great. There are call status codes known as Call Type (Missed, Incoming, Outgoing, ..) as well as a number Presentation code which is usually 1 (Allowed), although there are a few other values. Calls that show up on your phone as 'Private' numbers, will have presentation code 2 (Restricted). If you have enabled any call blocking features, then those show up too on blocked calls (known as block reason).</div>
<div>
<br /></div>
<div>
<b><span style="font-size: large;">Code to automate this parsing</span></b></div>
<div>
<b><br /></b></div>
<div>
A python script has been created to parse call log records from the <i>com.android.calllogbackup.data</i> file, <a href="https://github.com/ydkhatri/Android/blob/master/adb_backup_parser/callparser.py" target="_blank">available here</a>. The 010 template can be downloaded <a href="https://github.com/ydkhatri/010_Templates/blob/master/Android/android.data.calllogrecord.bt" target="_blank">here</a>.</div>
<div>
<br /></div>
<div>
<div>
<b><span style="font-size: large;">Forensic Gems - Detecting Deleted call records</span></b></div>
<div>
<br /></div>
<div>
Since each call record has a key which is the call id or the serial number of the call, I performed an experiment to see if deleting intermittent call records would change this number. It turns out that the number does not change, in effect allowing us to detect deleted call records. This is visible in the screenshot below, where you can see call IDs (serial numbers) of 1 through 8 but its missing 4 and 7. Those are the ones I had manually deleted from the Call logs on the phone through the available feature in the Phone app. This was also tested on a real phone with several hundred call records going back several months and it appears to hold true there too.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXrCMg14IAfOFU0cmHipbPQlmThQM8_GcCCGo79F55g9m_kV4Dg2CbArNMjeIkMQQPYiQbYn3-neYjyU_qPNASSGWOpW1ISGoNRJJNWpZA1EY3RPILWE6D4NGsajNcxwu_dAOp0cj-Qok/s1600/calllog.data.excel.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="158" data-original-width="934" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXrCMg14IAfOFU0cmHipbPQlmThQM8_GcCCGo79F55g9m_kV4Dg2CbArNMjeIkMQQPYiQbYn3-neYjyU_qPNASSGWOpW1ISGoNRJJNWpZA1EY3RPILWE6D4NGsajNcxwu_dAOp0cj-Qok/s1600/calllog.data.excel.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - Output of callparser.py, made pretty in Excel showing missing call ids.</i></td></tr>
</tbody></table>
<div>
This can be useful knowing that there can be records which are missing, perhaps intentionally.</div>
<div>
<br /></div>
<div>
<i>Stay tuned for Part 3, there is more good stuff in these key-value backups.</i></div>
<div>
<br /></div>
</div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-48700732124528866982019-10-07T15:26:00.000-04:002019-10-15T19:35:20.745-04:00ADB keyvalue backups and the .data format<div dir="ltr" style="text-align: left;" trbidi="on">
The ADB backup has been a very useful tool for getting data from Android phones, particularly those phones/devices that are otherwise not accessible due to lack of support by forensic software vendors or hardware/software issues with other methods.<br />
<br />
There is however one feature which I do not see being used by any of the vendors or FOSS or any other guides out there. I am specifically talking about about ADB's backup feature to backup key-value pairs. According to one <a href="https://www.xda-developers.com/android-oreo-adb-backup-better/" target="_blank">source</a>, since Oreo (8.0), the keyvalue backups are now available via adb backup.<br />
<br />
To get data with keyvalue pairs returned, you need to add the -keyvalue parameter to the adb backup command like. I like to use :<br />
<br />
<blockquote class="tr_bq">
<span style="background-color: yellow; font-family: "courier new" , "courier" , monospace;">adb backup -all -shared -system -keyvalue -f file.adb</span></blockquote>
<br />
Keyvalue backups give some very good information otherwise not available in the adb backup.<br />
<br />
<h4 style="text-align: left;">
So, where are the key-value backups located?</h4>
<br />
When viewing the adb tar archive, you will find one or more folders under each app's folder with names like <span style="font-family: "courier new" , "courier" , monospace;">k</span>, <span style="font-family: "courier new" , "courier" , monospace;">sp</span>, <span style="font-family: "courier new" , "courier" , monospace;">db</span>, .. The <span style="font-family: "courier new" , "courier" , monospace;"><b>k</b></span> folder holds the key value backup, having file(s) which ends in the extension <span style="font-family: "courier new" , "courier" , monospace;">.data</span>.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcZhydrTjpXSW4PBomcLDlMJbxHquoFaMruihjVVlz72Wg8oDx3FjMucWUX5OtPdVy-9f9E90_Zma05xPhbX5rozopnXgPBv500Yyu0_m9SoA37m2nu2fIkhOs32O2QzUtX8o4uIHeMNw/s1600/showing_k_folders.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="364" data-original-width="242" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcZhydrTjpXSW4PBomcLDlMJbxHquoFaMruihjVVlz72Wg8oDx3FjMucWUX5OtPdVy-9f9E90_Zma05xPhbX5rozopnXgPBv500Yyu0_m9SoA37m2nu2fIkhOs32O2QzUtX8o4uIHeMNw/s1600/showing_k_folders.JPG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Folders holding key-value .data files (this isn't all , there are many more)</i></td></tr>
</tbody></table>
The <span style="font-family: "courier new" , "courier" , monospace;">*.data</span> files are located in the <span style="font-family: "courier new" , "courier" , monospace;">k</span> folders usually having the same name as the package like <span style="font-family: "courier new" , "courier" , monospace;">com.android.calendar.data</span>.<br />
<br />
<h4 style="text-align: left;">
Parsing <span style="font-family: "courier new" , "courier" , monospace;">.data</span> files</h4>
This consists of a series of records, each starting with '<span style="font-family: "courier new" , "courier" , monospace;">Data</span>', and having a key (name) and value (data). The format is as follows, all data in this structure is stored as little-endian:<br />
<br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr>
<td style="background: #F4B083; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">Position<o:p></o:p></span></div>
</td>
<td style="background: #F4B083; border-left: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">Type<o:p></o:p></span></div>
</td>
<td style="background: #F4B083; border-left: none; border: solid #BFBFBF 1.0pt; mso-background-themecolor: accent2; mso-background-themetint: 153; mso-border-alt: solid #BFBFBF .5pt; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">Description<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">00<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: .85in;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">char[4]<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">‘Data’<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">04<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: .85in;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">uint<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">key_size<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">08<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: .85in;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">uint<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">data_size<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">12<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: .85in;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">char[key_size+1]<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">key_name<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">12 + key_size<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: .85in;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">char[]<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">pad to 4 byte boundary<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">..<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: .85in;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">char[data_size]<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">data<o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid #BFBFBF 1.0pt; mso-border-alt: solid #BFBFBF .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 76.25pt;" valign="top" width="102"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">..<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 74.7pt;" valign="top" width="106"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">char[]<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid #BFBFBF 1.0pt; border-left: none; border-right: solid #BFBFBF 1.0pt; border-top: none; mso-border-alt: solid #BFBFBF .5pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themeshade: 191; mso-border-left-alt: solid #BFBFBF .5pt; mso-border-left-themecolor: background1; mso-border-left-themeshade: 191; mso-border-right-themecolor: background1; mso-border-right-themeshade: 191; mso-border-themecolor: background1; mso-border-themeshade: 191; mso-border-top-alt: solid #BFBFBF .5pt; mso-border-top-themecolor: background1; mso-border-top-themeshade: 191; padding: 0in 5.4pt 0in 5.4pt; width: 171.0pt;" valign="top" width="228"><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in;">
<span style="font-family: "trebuchet ms" , sans-serif;">pad to 4 byte boundary</span><o:p></o:p></div>
</td>
</tr>
</tbody></table>
<br />
<br />
The value field can be different types depending on the data/database being backed up. It is different for different packages. You can find XML files, entire SQLITE databases in there, and also single byte true/false type settings.<br />
<br />
In the screenshot below, you can see the key-value records as parsed out for com.android.vending.data. The 010 template for this is available <a href="https://github.com/ydkhatri/010_Templates/blob/master/Android/Android.data.bt" target="_blank">here</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj61JwYZcv6gPaUbxOsCthf8-AclaSPgBNiAJ__xXNaPgh_KI5vf9PojcTKHeJ6BO_8fFlEMu72m7fk_igGImqdlzQsn0BzdYsGu9Ku88wSmp63SByS7moi0PT_tZVoAj_wlcr3NwA8m7Q/s1600/com.android.vending.data.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="574" data-original-width="604" height="608" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj61JwYZcv6gPaUbxOsCthf8-AclaSPgBNiAJ__xXNaPgh_KI5vf9PojcTKHeJ6BO_8fFlEMu72m7fk_igGImqdlzQsn0BzdYsGu9Ku88wSmp63SByS7moi0PT_tZVoAj_wlcr3NwA8m7Q/s640/com.android.vending.data.JPG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Hex editor view of com.android.vending.data, showing 'Data' records parsed out using an 010 template</i></td></tr>
</tbody></table>
<br />
In the above example, the value types are mostly True/False. But most other databases have other custom structures embedded there, which need further parsing.<br />
<br />
<i>In <a href="https://www.swiftforensics.com/2019/10/part-2-adb-keyvalue-backups-call-logs.html" target="_blank">part 2</a> of this ADB series, we explore the formats of call logs and other databases that are backed up. </i></div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com2tag:blogger.com,1999:blog-1264611260322778486.post-24861876160579911352018-12-29T11:52:00.001-05:002019-01-02T18:02:16.502-05:00Making NSKeyedArchives human readable<div dir="ltr" style="text-align: left;" trbidi="on">
If you've been doing macOS analysis, you are definitely familiar with the (<i>now not so new</i>) serialized plist format also known as an <b>NSKeyedArchive</b>. There are parsers available to extract data from this format, such as the <a href="https://github.com/cclgroupltd/ccl-bplist" target="_blank">ccl_bplist</a> from Alex Caithness (some more sophisticated ones listed at the bottom of this post). I've been using this library in mac_apt and other projects too. That is all old news.<br />
<div>
<br /></div>
<h4 style="text-align: left;">
Need for a human readable version</h4>
<div>
I always like to manually verify the results of my code by looking at the raw plist values to make sure my analysis programs parsed the right values as well as if they missed something. That isn't possible with NSKeyedArchives.</div>
<div>
<br /></div>
<div>
The ccl_bplist parser is great if you want to explore the structure of a plist interactively with python or programatically. What it does not do is auto-generate and save the deserialized version of that plist as a new human readable plist file. So that is what I set out to do yesterday. </div>
<div>
<div>
<br /></div>
</div>
<div>
Long story short, the code is available on github <a href="https://github.com/ydkhatri/MacForensics/tree/master/Deserializer" target="_blank">here</a>. </div>
<div>
<br /></div>
<div>
Below are screenshots showing an NSKeyedArchive sfl2 file (<i>Figure 1</i>) and its deserialized human readable form (<i>Figure 2</i>).</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhONqdc4bmPBMZ1V0KhpTQzdNrqy56chiobZipaz3ynknTPK_-2AeJXlhCBwK222dAY3BZz-Jr3PMloJ-4MowdDxsAk4Ebck5wlvfd0SqULWLH1IYsrD_FJ09YX1vpV6MX_A1upSGub9Tw/s1600/sfl2.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="794" data-original-width="626" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhONqdc4bmPBMZ1V0KhpTQzdNrqy56chiobZipaz3ynknTPK_-2AeJXlhCBwK222dAY3BZz-Jr3PMloJ-4MowdDxsAk4Ebck5wlvfd0SqULWLH1IYsrD_FJ09YX1vpV6MX_A1upSGub9Tw/s640/sfl2.PNG" width="504" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - NSKeyedArchive</i></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiadfIlelr3Te8GVra8dlbcOV2nG2p_rc3ZfOpiK-4loCfX6NzQrty3al4H11x7Wiz_j9_fciJYRZNacjIcjQN8LJgxqLWZf61tpQYiK14dmAEapLg55u-0Z4T4b62RPzFv3LxlPHnnoX0/s1600/deserialized_plist.PNG" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="473" data-original-width="715" height="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiadfIlelr3Te8GVra8dlbcOV2nG2p_rc3ZfOpiK-4loCfX6NzQrty3al4H11x7Wiz_j9_fciJYRZNacjIcjQN8LJgxqLWZf61tpQYiK14dmAEapLg55u-0Z4T4b62RPzFv3LxlPHnnoX0/s640/deserialized_plist.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Deserialized form of NSKeyedArchive</i></td></tr>
</tbody></table>
<br />
<div>
<br /></div>
<div>
Using <a href="https://github.com/cclgroupltd/ccl-bplist" target="_blank">ccl_bplist</a> and <a href="https://github.com/wooster/biplist" target="_blank">biplist</a>, this should have been a 2-5 line program, ccl_bplist to deserialize and biplist to write out the new plist. However it turned out to be a few lines more than that as I had to write a short recursion function to process the plist data because the ccl_bplist also includes <i>$class</i> information which needed to be stripped out. Also as I found out, this will only work with Python 3 because Python 2 does not distinguish between the types <i>str </i>and <i>bytes</i>. In Python 2, all these (below) are the same:</div>
<div>
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> '\x12\x34'</span></div>
<div>
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> b'\x12\x34'</span></div>
<div>
<span style="color: purple; font-family: "courier new" , "courier" , monospace;"> bytes('\x12\x34') </span></div>
<div>
<br /></div>
<div>
This creates a problem as biplist cannot distinguish between binary blobs and strings as they all appear to be strings and it fails when it encounters a byte that cannot be encoded as a string. No problems with python 3 as these are different distinct types there.</div>
<div>
<br /></div>
<div>
The code is <a href="https://github.com/ydkhatri/MacForensics/tree/master/Deserializer" target="_blank">here</a>.<br />
<br /></div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Alternatives</h4>
<div>
Since writing this, I did find another library (<a href="https://github.com/xa4a/bpylist2/" target="_blank">bpylist2</a>) that has similar functions, reading/writing binary plists as well as creating/reading keyed archives. I haven't tested it yet.<br />
<br />
<b><i>(Update)</i></b> Our good friends at Aon also maintain a library for plist parsing which seems pretty solid and worth checking out <a href="https://github.com/strozfriedberg/plistutils/blob/master/plistutils/nskeyedarchiver.py" target="_blank">here</a>.</div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com2tag:blogger.com,1999:blog-1264611260322778486.post-6280813398825432412018-11-24T12:27:00.000-05:002018-11-24T12:55:33.138-05:00$Recycle bin and Undo operations<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSKk_Xn2ftnifstmjJT7mUNlKFdMNl08NoZ2aqRTuHxhb4uqlipcTw-8DkUm3gioilZRzB8ARzfoaNU2eMowgS3MfXQCfQ4IzyleDsJ25wEVX_p7oHDEG9osgy8HBOQVYwzeRWnHyCsZo/s1600/recycle_bin.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="398" data-original-width="340" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSKk_Xn2ftnifstmjJT7mUNlKFdMNl08NoZ2aqRTuHxhb4uqlipcTw-8DkUm3gioilZRzB8ARzfoaNU2eMowgS3MfXQCfQ4IzyleDsJ25wEVX_p7oHDEG9osgy8HBOQVYwzeRWnHyCsZo/s200/recycle_bin.png" width="170" /></a></div>
<br />
This week Phil Moore made an excellent finding (<a href="https://thinkdfir.com/2018/11/23/quick-post-notes-on-the-win10-recycle-bin/" target="_blank">link here</a>), one that most of us have seen for years but not investigated. Those $I files that seem orphaned/abandoned without explanation now have one. Phil notes that every time a file/folder is deleted and then restored, the $I file stays behind in the recycle bin. This post is about some follow up testing and results.<br />
<br />
I did some quick tests - deleting and restoring files in a few different ways testing $I & $R file creation every time. I'm not even looking at indexes or timestamps, just file creation/deletion. My testing was on Windows 10 (32bit) version 10.0.10586.106<br />
<br />
I'm not reproducing all the output here, but only the summary. Here are a few ways to send files to the recycle bin. Remember you cannot do any operations using the command line (as that operates directly at the file system level and does not use the recycle bin abstraction).<br />
<br />
First we delete a file (right-click and select <i>Delete</i>). Let's try to restore now using any one of the following ways:<br />
<br />
1. Right-click the file, click <i>Restore</i> on file in the recycle bin<br />
2. <i>Cut</i> file from recycle bin and <i>Paste</i> elsewhere<br />
3. Drag the file from recycle bin and into another folder<br />
4. Undo the last operation using Ctrl-Z OR right-click & select <i>Undo Delete</i><br />
<i><br /></i>
<i></i><br />
<b>Only the 4th method (<i>Undo Delete</i>) results in deletion of the $I file</b>. The other methods leave it behind.<br />
<div>
<br /></div>
<div>
I believe this to be a windows bug. Interestingly, when you Restore a file from the bin (right-click & <i>Restore</i>), then right-click on the desktop (or in any folder), the context menu has an new item called '<i>Undo Move</i>'. It does NOT say '<i>Redo Delete</i>'. For every other action, you will see a '<i>Redo ACTION</i>' in the menu (see figure 2). So I believe when you restore a file, windows just performs a file <i>move</i> (on the $R file) and thus marked it as a <i>move</i> in the last performed action. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaaf7GWpGU9vO9gI5vtZK4YRu4omaQ48r3RiRk-yLJQTH3-w7PPddhW1tvNCSlNzM8a47IWlDfCyrZKi9jRvOgKnDZAwP3rM-SgainIFQCILiAfbN3wHKcvWvsrAU5w2JvpI6IxPnRVF8/s1600/undo_move_menu.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="590" data-original-width="486" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaaf7GWpGU9vO9gI5vtZK4YRu4omaQ48r3RiRk-yLJQTH3-w7PPddhW1tvNCSlNzM8a47IWlDfCyrZKi9jRvOgKnDZAwP3rM-SgainIFQCILiAfbN3wHKcvWvsrAU5w2JvpI6IxPnRVF8/s200/undo_move_menu.png" width="164" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - 'Undo Move' seen after restoring file</i></td></tr>
</tbody></table>
<div>
Now, clicking on the <i>Undo Move</i> will result in sending the file back to the recycle bin but as a new $I & $R pair, as if it was an entirely new delete operation. </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVEzdeeV66oDw-CoWiMhyphenhyphen7N6qAnZk8U5B78ZsD4oVs6saY084fZUnw9nTOPBe6TzrT1JEljeobIB5BaGAFrlypl6r5d_uYAz0GhOfOW9h5JpngsF5MNU2XNI5VK9SWlvfDrbNht9P2JYc/s1600/undo_rename.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="638" data-original-width="488" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVEzdeeV66oDw-CoWiMhyphenhyphen7N6qAnZk8U5B78ZsD4oVs6saY084fZUnw9nTOPBe6TzrT1JEljeobIB5BaGAFrlypl6r5d_uYAz0GhOfOW9h5JpngsF5MNU2XNI5VK9SWlvfDrbNht9P2JYc/s200/undo_rename.png" width="152" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - 'Redo Rename' seen after using 'Undo Rename' from an unrelated file rename operation</i></td></tr>
</tbody></table>
<div>
In figure 3 below, you can see multiple abandoned $I files after doing this multiple times (delete, then restore).</div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVu4WkNepxxRgUrkgDWt1ewtQ0atQhZ_aChEo2H-5uS3Lp9WtM6kR_QsPKQyx7Kct9MSoZVIXVDZ2ieZTqUWDzMElI_AKtIxPU7ijjHCDCEuGODBoXCnLxd6iZklejvC8ktU37hJ0v7Vs/s1600/recycle_bin_dir.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="706" data-original-width="1152" height="392" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVu4WkNepxxRgUrkgDWt1ewtQ0atQhZ_aChEo2H-5uS3Lp9WtM6kR_QsPKQyx7Kct9MSoZVIXVDZ2ieZTqUWDzMElI_AKtIxPU7ijjHCDCEuGODBoXCnLxd6iZklejvC8ktU37hJ0v7Vs/s640/recycle_bin_dir.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td class="tr-caption" style="font-size: 12.8px;"><i>Figure 3 - Recycle bin folder after multiple deletes and restores showing abandoned $I files</i></td></tr>
</tbody></table>
</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
Let us know (comments or twitter) if you know of more ways to delete/restore.</div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com2tag:blogger.com,1999:blog-1264611260322778486.post-80641817110016658442018-11-06T23:14:00.003-05:002018-11-07T09:55:47.902-05:00The ._ (dot-underscore) file format<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
If you've ever looked at removable media and found several hidden files which start with ._ and there exists one for almost every file (or folder) on the disk, this is the result of having that media being used on macOS.</div>
<div>
<br /></div>
macOS keeps a copy of file metadata in a separate area, known as Extended Attributes (xattr) on HFS+ or APFS. However, when writing to external media which is not formatted as HFS+ or APFS (thus not having the capability to store extended attributes), it will write this information out as a separate file which will have the same name, just prefixed with <i>dot-underscore </i> <b>._</b> as seen in the screenshot below.<br />
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNMhqCQFXLWI3numXM4Y5cRybV0arIjmBKihwizqGF_ekUivAxCyzuk7f_-6hGlbX1NrdnaabnNujb_VJPNtCZvEfRJk0dYNWu_5j_wy85BDz2Uj97ECTl0jHcMJXtffAnUsO3ElxwqVU/s1600/Showing+._+files.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="180" data-original-width="544" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNMhqCQFXLWI3numXM4Y5cRybV0arIjmBKihwizqGF_ekUivAxCyzuk7f_-6hGlbX1NrdnaabnNujb_VJPNtCZvEfRJk0dYNWu_5j_wy85BDz2Uj97ECTl0jHcMJXtffAnUsO3ElxwqVU/s1600/Showing+._+files.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Screenshot showing exFAT volume on External USB disk</i></td></tr>
</tbody></table>
<div>
While this is well known for many years, this information is often overlooked in a forensic investigation. On media that has interacted with both macOS and Windows (or even Linux), macOS will create these files and delete them too when the original file is deleted. <u>However, if the file is deleted or renamed on Windows or Linux, then the dot-underscore files will be left behind untouched</u>. A while back, Lee Whitfield touched upon this <a href="https://forensic4cast.com/2016/10/macos-file-movements/" target="_blank">here</a> specifically pointing out its use for knowing the date & time that a file was copied onto the media. However, there is useful information inside the file too. </div>
<div>
<br /></div>
<div>
This file can contain useful metadata such as kMDItemWhereFroms (<i>URL of file if downloaded from internet</i>) and kMDItemDownloadedDate (<i>Date & Time when it was downloaded</i>) among other extended attributes.</div>
<div>
<br /></div>
<div>
After a bit of reverse engineering, I wrote an 010 hex editor template to parse this information, it is available at <a href="https://github.com/ydkhatri/MacForensics/blob/master/DotUnderscore_macos.bt">https://github.com/ydkhatri/MacForensics/blob/master/DotUnderscore_macos.bt</a>. </div>
<div>
<br /></div>
<div>
In the screenshot below, you can see it being run on one such file. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh85P3yObKDH3IZoMyb6ZoON325zECmmGdQneqTxAAeMmQgZdixcmhcP-0riIwB_9o-5M-95TEG03e49_4tQy24uFjrISjwVtD1shBA1_8fWkc2HEqmetXu-XsmqOIb_r_U1flxWmsLbO8/s1600/dotUnderscoreTemplateRun.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="762" data-original-width="1108" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh85P3yObKDH3IZoMyb6ZoON325zECmmGdQneqTxAAeMmQgZdixcmhcP-0riIwB_9o-5M-95TEG03e49_4tQy24uFjrISjwVtD1shBA1_8fWkc2HEqmetXu-XsmqOIb_r_U1flxWmsLbO8/s640/dotUnderscoreTemplateRun.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - DotUnderscore_macOS.bt template output</i></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
Here is analysis from the data extracted:</div>
<div>
<br /></div>
<div>
<br />
<table cellpadding="2" cellspacing="0" style="border: 1px solid rgb(221, 221, 221);"><tbody>
<tr>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><b>Attribute Name</b></td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><b>Value</b></td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><i><b>Meaning</b></i></td>
</tr>
<tr>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><b>kMDItemWhereFroms</b></td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);">https://upload.wikimedia.org/wikipedia/commons/3/3c/Thiruvalluvar_Statue_at_Kanyakumari_beach.jpg</td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><i>URL from where it was downloaded</i></td>
</tr>
<tr>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><b>kMDItemDownloadedDate</b></td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);">0x41BFB51D1CFFA4F8 (11/09/2017 23:32:44)</td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><i>Timestamp when file was downloaded</i></td>
</tr>
<tr>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><b>com.apple.quarantine</b></td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);">0083;5a04e59c;Safari;A451620D-2B49-49BD-ADC1-88DEBEA66582</td>
<td style="border-bottom: 1px solid rgb(221, 221, 221); border-right: 1px solid rgb(221, 221, 221);"><i>File was downloaded using the Safari browser*</i></td>
</tr>
</tbody></table>
</div>
<div>
<i><br /></i></div>
<div>
<i>kMDItemDownloadedDate is stored in a plist as a date value, which is stored as a 64-bit double that is the number of seconds since 01/01/2001</i></div>
<div>
<i>The template does not parse the plist for you, you can export it out and open in any plist viewer to view the human-readable date value.</i><br />
<i><br /></i>
<i>*The 3rd item in com.apple.quarantine's value (separated by ;) is the Application (agent) name which downloads the file. For more details on this, read Howard Oakley's <a href="https://eclecticlight.co/2017/08/15/quarantined-more-about-the-quarantine-extended-attribute/" target="_blank">blog post</a>. </i></div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com1tag:blogger.com,1999:blog-1264611260322778486.post-56196449065344760102018-10-16T00:30:00.003-04:002018-10-16T11:18:32.520-04:00The user spotlight database<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
On macOS, the spotlight database is a central database holding metadata of all files/folders that macOS indexes and is always located at the root of any disk under <span style="background-color: yellow; font-family: "courier new" , "courier" , monospace;">/.Spotlight-V100</span>. </div>
<div>
<br /></div>
<div>
However, while browsing the folders on my macOS 10.14 (Mojave) image, I find a folder that contains yet another spotlight database. It appears that there are now more than one spotlight databases on a single disk. There is one for each user located at:</div>
<blockquote class="tr_bq">
<span style="background-color: yellow;"><span style="font-family: "courier new" , "courier" , monospace;"><b>~/Library/Metadata/CoreSpotlight/index.spotlightV3/</b></span></span></blockquote>
<div>
As with the other spotlight database, the files that hold the information are <span style="background-color: yellow;"><span style="font-family: "courier new" , "courier" , monospace;">store.db</span></span> and <span style="background-color: yellow; font-family: "courier new" , "courier" , monospace;">.store.db</span><span style="font-family: inherit;"><span style="background-color: white;">.</span></span></div>
<div>
<br /></div>
<div>
Mojave (10.14) isn't the first version of macOS to include this database. This appeared first in High Sierra (10.13).</div>
<div>
<br /></div>
<h3 style="text-align: left;">
What is in it?</h3>
<div>
The per-user database store is used to store metadata from items that aren't files or folders. Items seen so far are:</div>
<div>
<ul style="text-align: left;">
<li>Safari browser history (web pages visited)</li>
<li>Safari browser bookmarks</li>
<li>News App history (web pages visited)</li>
<li>Notes App notes</li>
<li>Maps App data (locations?)</li>
</ul>
I would speculate that emails would also be seen here, as a number of email related fields are present too:<br />
<ul style="text-align: left;">
<li>kMDItemRecipientEmailAddresses</li>
<li>kMDItemPrimaryRecipientEmailAddresses</li>
<li>kMDItemAdditionalRecipientEmailAddresses</li>
<li>kMDItemHiddenAdditionalRecipientEmailAddresses</li>
</ul>
</div>
<div>
<br /></div>
<div>
However no email metadata was seen with a single configured IMAP account in the Mail app.<i> </i><br />
<i><br /></i>
<i>Since this is a test environment with very little activity and almost no apps other than those that come with macOS, there is likely to be a lot more metadata from different apps in this database on real world systems.</i></div>
<div>
<br /></div>
<h3 style="text-align: left;">
Why a separate database?</h3>
<div>
Reading Apple's documentation <a href="https://developer.apple.com/documentation/corespotlight">here</a> and <a href="https://developer.apple.com/design/human-interface-guidelines/macos/system-capabilities/search-and-spotlight/">here</a> seems to suggest that this is the implementation of functionality intended to allow app developers to provide in-app content searches and includes the ability to define metadata to do so. Items indexed are not required to be files.</div>
<div>
<br /></div>
<div>
Prior to this (10.12 and below) there existed a folder at</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><b>~/Library/Caches/Metadata/Safari/</b></span></div>
<div>
<br /></div>
<div>
which housed all the *.webhistory files. See pic below.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK_ricyosZ_bciVlSoOZeYNINVJK1kOS_FifutUC8-tqQbJH99xzvtL47nrba26rLMML7B2ZULCbGO_P9hpvGtb-aAke5Pbmw0iBQgMK1DYtexs3YPRUBNkUUYbLoBcvAyRyCFO-bS3Sc/s1600/safari+history+as+files.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="430" data-original-width="1600" height="171" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK_ricyosZ_bciVlSoOZeYNINVJK1kOS_FifutUC8-tqQbJH99xzvtL47nrba26rLMML7B2ZULCbGO_P9hpvGtb-aAke5Pbmw0iBQgMK1DYtexs3YPRUBNkUUYbLoBcvAyRyCFO-bS3Sc/s640/safari+history+as+files.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - webhistory files in ~/Library/Caches/Metadata/Safari/</i></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
The individual files were plists which were then indexed by spotlight.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDx9wFgDnAI65BCQt4uUI46Sz8H6fG_vkzy6rBkqIN_QypOHutwvcuednoD7St_9mvLEmaQN3WuiL0TFM8I9u2TS8ax76lxi5Ig1gDPEPbRq95VPXI-WRkBJbMrvrsHUlmT4nQyCUfPZk/s1600/webhistory.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="250" data-original-width="1598" height="99" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDx9wFgDnAI65BCQt4uUI46Sz8H6fG_vkzy6rBkqIN_QypOHutwvcuednoD7St_9mvLEmaQN3WuiL0TFM8I9u2TS8ax76lxi5Ig1gDPEPbRq95VPXI-WRkBJbMrvrsHUlmT4nQyCUfPZk/s640/webhistory.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - webhistory file content</i></td></tr>
</tbody></table>
<div>
<i>That folder now does not exist and in its place we have the new spotlight database.</i><br />
<i><br /></i></div>
<h3 style="text-align: left;">
Parsing the data</h3>
<div>
Using mac_apt's single_plugin script to only run the spotlight plugin over individual store.db files, we can easily parse the database.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWpL3tEkqUGzCeduwaX2pQQCFk0Dw28SLp7Dw9uK0gezScl1znnhNEvMoU9rEYnVSDuJdrrDRuRXMJ0g3Eei4BDQBeTpM503HSTm7fHhdCBLo3fSz0ZEx2syMQv4AZ7hrMPUFbKUgLoPA/s1600/mac_apt_singleplugin-spotlight.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="374" data-original-width="764" height="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWpL3tEkqUGzCeduwaX2pQQCFk0Dw28SLp7Dw9uK0gezScl1znnhNEvMoU9rEYnVSDuJdrrDRuRXMJ0g3Eei4BDQBeTpM503HSTm7fHhdCBLo3fSz0ZEx2syMQv4AZ7hrMPUFbKUgLoPA/s640/mac_apt_singleplugin-spotlight.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - mac_apt_singleplugin to parse the store.db file</i></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
There are quite a few fields, some more important than others. Below are some screenshots showing selected data (notes, safari history, and news). mac_apt gives you the data as a spreadsheet, sqlite db and a flat text file too (similar to <i>mdls </i>output).</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUp2ga-R_kRColRkRwu9yerZtGyTsLtBZ5gV0cjxwrmk4HCNDFhC-7W-3FaYEFUzlnq-6Wm7QT8ibieqciPjXbLf4zHOAw6Vtto4cbOKEzPaQk9vVAbvxnGMXpff0eDSmbqPViOaWI5s0/s1600/Notes-spotlight2.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="296" data-original-width="1233" height="152" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUp2ga-R_kRColRkRwu9yerZtGyTsLtBZ5gV0cjxwrmk4HCNDFhC-7W-3FaYEFUzlnq-6Wm7QT8ibieqciPjXbLf4zHOAw6Vtto4cbOKEzPaQk9vVAbvxnGMXpff0eDSmbqPViOaWI5s0/s640/Notes-spotlight2.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 4 - notes metadata from store.db (not all fields shown here)</i></td></tr>
</tbody></table>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSllB5ZILrKihcC6hV09cx-fFwtTgNYEVPtMbVbeTlKqI1LBLxCNQu1YMTM-E5jVaY8BbBjGaB5FtDLpeRjIZnaHGFvakmGzz3to4iMxF_dj8F9iHiX94qypiohMMWx_ePdNgmATAq1V8/s1600/safari-history-spotlight2.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="402" data-original-width="1341" height="190" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSllB5ZILrKihcC6hV09cx-fFwtTgNYEVPtMbVbeTlKqI1LBLxCNQu1YMTM-E5jVaY8BbBjGaB5FtDLpeRjIZnaHGFvakmGzz3to4iMxF_dj8F9iHiX94qypiohMMWx_ePdNgmATAq1V8/s640/safari-history-spotlight2.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 5 - safari history from store.db (not all fields shown here)</i></td></tr>
</tbody></table>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinp1NtZCFcpCn6gMBi9pu-Z1lBu5T9ScKoaH8LMt_-wSHIdNw0RKtpgzDWF2uxlHY6fZaQV8O4Y_i2fSKBkbvqn9sJCI_9Hw4lXeQITOEp6g_Gv3bnDwxKpFPlg8wnVQFNR7VCno0u_Dw/s1600/news-history-spotlight.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="486" data-original-width="732" height="424" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinp1NtZCFcpCn6gMBi9pu-Z1lBu5T9ScKoaH8LMt_-wSHIdNw0RKtpgzDWF2uxlHY6fZaQV8O4Y_i2fSKBkbvqn9sJCI_9Hw4lXeQITOEp6g_Gv3bnDwxKpFPlg8wnVQFNR7VCno0u_Dw/s640/news-history-spotlight.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 6 - A single entry from News app showing all metadata (parsed from store.db)</i></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<i><a href="https://github.com/ydkhatri/mac_apt">mac_apt's</a> spotlight plugin has been updated to automatically handle/process these user spotlight databases now. </i></div>
<div>
<br /></div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-29038221172466646552018-08-21T23:40:00.003-04:002018-08-21T23:43:42.478-04:00An open source spotlight parser<div dir="ltr" style="text-align: left;" trbidi="on">
Spotlight is the name of the indexing system which comes built into macOS. It is responsible for continuous indexing of files and folders on all attached volumes. It keeps a copy of all metadata for almost every single file and folder on disk.<br />
<br />
Thus, it can provide some excellent data for your investigation. While much of the same information can be obtained if you have access to the full disk image, it is known that there is information in this database that is not available elsewhere. Details like Date(s) Last Opened or Number of Times (an application or file) is Opened/Used are not available anywhere else on the file system. Unfortunately though it uses a proprietary undocumented format, and no publicly available code existed to read it. So over the last few months, I’ve been studying the file format of these databases and have created a tool/library to read and extract the data contained within.<br />
<br />
The library and tool are open sourced now and located here:<br />
<a href="https://github.com/ydkhatri/spotlight_parser">https://github.com/ydkhatri/spotlight_parser</a><br />
<br />
The format of the database will be discussed in a later post.<br />
<br />
For those familiar with macOS, you know this data (contained in the database) can be obtained on a locally mounted volume using the macOS built-in <i>mdls </i>utility. However to do this, you need to mount your disk image on a mac to do so and the utility can only be run on individual files/folders, not the entire disk. It can be run recursively (with a bit of command line fu) on the entire volume but the output is not easy to read then.<br />
<br />
If you don't prefer to do that, run spotlight_parser instead. Just point it to the database files which are named <i>store </i>and <i>.store </i>(located in the /.Spotlight-V100/Store-V2/<<i>UUID</i>> folder) and let it parse out the complete database for you.<br />
<br />
Here is a screenshot of spotlight_parser running. Depending on how much data is contained in the database, this can take anywhere between a few seconds to a few minutes (5-10 on very large disks with lots of files).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4Vm9U8B0oHQEA_tGSjd-PxAilPeCrInStAkpU1QhHcnZfxUnRteDMkQDEKQw2XIkSOIXyD4J6T7KTT5jDQ_e6XmlY2Ldwwa7fUA0MI1l5FdECHuesqgpQDgKSeGNtqSZGdvKH4AYwu3Y/s1600/spotlight_parser_run.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="353" data-original-width="818" height="276" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4Vm9U8B0oHQEA_tGSjd-PxAilPeCrInStAkpU1QhHcnZfxUnRteDMkQDEKQw2XIkSOIXyD4J6T7KTT5jDQ_e6XmlY2Ldwwa7fUA0MI1l5FdECHuesqgpQDgKSeGNtqSZGdvKH4AYwu3Y/s640/spotlight_parser_run.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Running spotlight_parser</i></td></tr>
</tbody></table>
<br />
Once done, you will have 2 files as output. One is a text file (<i>prefix_</i>data.txt) containing the database dump of all entries. The other is a CSV (actually tab separated) which tries to build a path for every file/folder using inode number (CNID) from data available in the database. Since not every single folder may be included, some paths may not resolve and you might get <b>..NOT FOUND..</b> in the path sometimes along with an error on the console as seen above.<br />
<br />
In the <i>prefix_</i>data.txt file, you will see some XML content (configuration information) at the beginning followed by database entries for files and folders.<br />
Below is a snippet of the <i>prefix_</i>data.txt file, showing only output for a single jpg image file.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4gzux2Vv8dZEK7Uqn9jRa2liSvwTFIQvmb5saqhXUa5W266XarsjJOz9GITRK8Zfh50igZVWHEapFSzKdgkyjL1To0VKNcV7gC_RrHFG5SQUADUIienqnuXCTTDt5TZ3g7F8mdpuYHd4/s1600/spotlight_script_output.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="711" data-original-width="883" height="513" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4gzux2Vv8dZEK7Uqn9jRa2liSvwTFIQvmb5saqhXUa5W266XarsjJOz9GITRK8Zfh50igZVWHEapFSzKdgkyjL1To0VKNcV7gC_RrHFG5SQUADUIienqnuXCTTDt5TZ3g7F8mdpuYHd4/s640/spotlight_script_output.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Output showing a single jpg file's metadata information from database</i></td></tr>
</tbody></table>
<br />
Here the text in <span style="color: red;">Red </span>is metadata pertaining to a single entry in the database, including the date and time it was last updated. This is followed by the metadata itself. The items in <span style="color: #3d85c6;">Blue</span> are information only available in the spotlight database. The last two may be of particular interest to an investigator.<br />
<i><br /></i>
<i>Note - The screenshot above is not from any special version of the code, actual output is plain text, it has no coloring! Colors were added just for explanation.</i><br />
<br />
The spotlight_parser has been incorporated into <a href="https://github.com/ydkhatri/mac_apt">mac_apt</a> as the SPOTLIGHT plugin. In mac_apt, the output is also available in an sqlite database making it easier to query. Mark McKinnon has forked a version of this library and also added sqlite capability, it is available <a href="https://github.com/markmckinnon/spotlight_parser">here</a>.<br />
<br />
While this exposes the data and makes it available, it is still not easy to query. Perhaps one of these days, I will write a GUI application with drop-down boxes for easily accessing and querying the output data.<br />
<br /></div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-71425703576250250072018-07-20T10:45:00.001-04:002018-08-21T23:41:24.974-04:00APFS template for 010 Editor<div dir="ltr" style="text-align: left;" trbidi="on">
For quite some time, I've been analyzing APFS mostly with custom python code, which is not very efficient and rather time consuming and is not visual. Since most people doing any kind of serious hex editing use the <a href="http://sweetscape.com/010editor/">010</a> Editor (as do I), this was long overdue.<br />
<br />
I've created an 010 template, which is basically a port from the <a href="https://github.com/cugu/apfs.ksy/">apfs.ksy project</a>. This has taken quite a bit of time and I hope you find it useful. Not all structures are known, there are some parts that may be incorrect. This is a work in progress as more details about APFS emerge..<br />
<br />
Link: <a href="https://github.com/ydkhatri/APFS_010/blob/master/apfs.010.bt">https://github.com/ydkhatri/APFS_010/blob/master/apfs.010.bt</a><br />
<br />
The template will not parse out the file system tree yet. With APFS this is challenging to do within 010's template capabilities as you cannot create local objects or classes and/or store temporary objects. The template does however define most of the structures and will follow most pointers (to other disk blocks and parse them) automatically when you start expanding the structures in the template viewer.<br />
<br />
To use the template, simply load your APFS image (unencrypted only) into 010. Then edit the template to set the <i>Apfs_Offset</i> variable to the byte offset of wherever your APFS partition starts. Now run the template. The APFS start offset can be located easily by running the GPT template (which you can find on 010's website or in the program's template repository). The GPT template will give you the sector offset, multiply it by sector size (usually 512 or sometimes 4096) to get the byte offset (location) of the APFS partition.<br />
<br /></div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-61508226400038661632018-05-03T14:59:00.002-04:002018-08-21T23:42:01.964-04:00Bash sessions in macOS (and why you need to understand its working)<div dir="ltr" style="text-align: left;" trbidi="on">
While all versions of macOS have provided <a href="https://www.cnet.com/news/using-and-managing-the-terminal-history-in-os-x/">bash_history</a> for users, since macOS 10.11 (El Capitan), we get even more information on terminal history through the bash sessions files. This is not a replacement for the old .bash_history file which is still there.<br />
<br />
There are several problems with bash_history - you cannot tell when any command in that file was run, the sequence of commands may not be right, and so on. For more on that, refer Hal Pomeranz's excellent talk - <a href="http://www.deer-run.com/~hal/DontKnowJack-bash_history.pdf">You don't know jack about Bash history</a><br />
<br />
Even if there were no anomalies and only a single terminal was always in use, there is still the issue of how do I know which command was run when? With Bash sessions, macOS gives us more data to work with. Since El Capitan, every new terminal window will be tracked independently with a <i>TERM_SESSION_ID</i> which appears to be a randomly generated UUID.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYbXMZ-VRziXBezoKjHfqb8-WSLPJ4XpMX_8-719q8dT0sh0rvoi0zchx5yw3oIyfFWYqznGWf9kQ-uZFjsoDsuZZQVjDbQwY0_MawcLLNK-eT4GsBFLBlzGWZ9WQyFWpzQpK54JwfNP8/s1600/get_session_id.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="99" data-original-width="425" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYbXMZ-VRziXBezoKjHfqb8-WSLPJ4XpMX_8-719q8dT0sh0rvoi0zchx5yw3oIyfFWYqznGWf9kQ-uZFjsoDsuZZQVjDbQwY0_MawcLLNK-eT4GsBFLBlzGWZ9WQyFWpzQpK54JwfNP8/s1600/get_session_id.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1 - Fetching terminal's session id</i></td></tr>
</tbody></table>
<br />
Each session can also be restored when you shutdown and restart your machine with the <i>"Reopen windows when logging back in"</i> option set. Perhaps for this purpose, session history (a subset of bash history) is tracked and saved separately on a per session basis.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1jvmc9jY3CIeU9QtePxZkmGu3Kx_nDghPYcHFhuRzmAwyHiW1XUhngt0GGQwXPOQBCEt0mj5MDlHQXXlwR5f19IKs4JQWAo_1BjQ6ZODAoBflbqsOawHjVmSHcuwvw1ZB5P9OKXZ6cd8/s1600/restored_session_window.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="100" data-original-width="434" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1jvmc9jY3CIeU9QtePxZkmGu3Kx_nDghPYcHFhuRzmAwyHiW1XUhngt0GGQwXPOQBCEt0mj5MDlHQXXlwR5f19IKs4JQWAo_1BjQ6ZODAoBflbqsOawHjVmSHcuwvw1ZB5P9OKXZ6cd8/s1600/restored_session_window.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2 - Restored session</i></td></tr>
</tbody></table>
<br />
<h3 style="text-align: left;">
Show me the artifacts!</h3>
The location you want to go to is <i style="background-color: yellow;">/Users/<USER>/.bash_sessions</i><br />
<br />
You will find 3 files for each session as seen in screenshot below.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpkxoHQ-KCEIUNMBYj2CGrGssjLIQoOE9n6GVfJM77vY0cjkUMep2LTUf3pF4tpRTSC3mDg1AKlZBRuZDZ1BNzft2niCi5E5yb1IMDKRslD5r6DNg9vhMGJ8LBb27ojkgOOvBTi6v1uwc/s1600/ls_bash_sessions.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="236" data-original-width="768" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpkxoHQ-KCEIUNMBYj2CGrGssjLIQoOE9n6GVfJM77vY0cjkUMep2LTUf3pF4tpRTSC3mDg1AKlZBRuZDZ1BNzft2niCi5E5yb1IMDKRslD5r6DNg9vhMGJ8LBb27ojkgOOvBTi6v1uwc/s640/ls_bash_sessions.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 3 - .bash_sessions folder contents</i></td></tr>
</tbody></table>
<br />
<br />
<b style="background-color: yellow;"><span style="font-family: "courier new" , "courier" , monospace;">TERM_SESSION_ID.history --> </span>Contains session history</b><br />
<b style="background-color: yellow;"><span style="font-family: "courier new" , "courier" , monospace;">TERM_SESSION_ID.historynew --> </span>Mostly blank/empty</b><br />
<b style="background-color: yellow;"><span style="font-family: "courier new" , "courier" , monospace;">TERM_SESSION_ID.session --> </span>Contains the last session resume date and time</b><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT-Xt_Pt9ItyKy9dTUNHI5BDrzdvWuBvYrn8mlcQC0nmNpvDGo4E4R3m_Pj0GOeH_h6qrD1DiYDxbq1WrQeSpMtwqOBGoPIi1SluiapA9kPPBGLQCyIb53QqzqM2-H85YK-ATw3Wm0Rnc/s1600/session+file.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="68" data-original-width="445" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT-Xt_Pt9ItyKy9dTUNHI5BDrzdvWuBvYrn8mlcQC0nmNpvDGo4E4R3m_Pj0GOeH_h6qrD1DiYDxbq1WrQeSpMtwqOBGoPIi1SluiapA9kPPBGLQCyIb53QqzqM2-H85YK-ATw3Wm0Rnc/s1600/session+file.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 4 - Sample .session file</i></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqWuSBKnTKgJbSa3TjqR9yTFnOmiWlXyi_LgDSgsDMFaJxKxLSYUD7gVpxmv-pbSTB4-XUuyCUt2N_by71js_KcKphlm-ikyD5NUXRAC3cZq73OS53YiMkTAURbripv7YdSu_9kb5MJtw/s1600/history+file.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="221" data-original-width="474" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqWuSBKnTKgJbSa3TjqR9yTFnOmiWlXyi_LgDSgsDMFaJxKxLSYUD7gVpxmv-pbSTB4-XUuyCUt2N_by71js_KcKphlm-ikyD5NUXRAC3cZq73OS53YiMkTAURbripv7YdSu_9kb5MJtw/s1600/history+file.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 5 - Sample .history file showing commands typed at terminal </i></td></tr>
</tbody></table>
<br />
<br />
<h3 style="text-align: left;">
How this helps?</h3>
Some (but not all) of the problems associated with reading .bash_history are now gone.<br />
Theoretically, as bash history is now also stored on a per session basis, this should make it trivial to track commands run in different windows (sessions). If you were expecting history for a single session in its <i>.history</i> file, then you thought wrong. The <i>.history</i> file contains all previous history (from earlier sessions) and then appended at the very end, the history for this session.<br />
<br />
So can we reliably break apart commands per session? Is the sequence of commands intact? Let's run a small experiment to find out.<br />
<br />
We create two sessions (2 terminal windows) and run a few commands in each session. Commands are interspersed, so we run a command in <i>Session-1</i>, then another in <i>Session-2</i> and then again something in <i>Session-1</i>. We will try to see if order is maintained.<br />
<br />
<i>Session-1</i> started 9:44<br />
<i>Session-2</i> started 9:51<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSDJNxJyNi8Zextky5dUVZSE4U3amsrJkJRICFk_OR1hu2qL4XLrg-WiTfvgN9EGtzsqKziRB3l0cKQ-GSXCSDvM8OGC2eFxa_JOEnMnXw8wcYRd-k-uEcNseth1D-STGh2rN5NRUD2X0/s1600/session+commands+sequence.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="198" data-original-width="292" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSDJNxJyNi8Zextky5dUVZSE4U3amsrJkJRICFk_OR1hu2qL4XLrg-WiTfvgN9EGtzsqKziRB3l0cKQ-GSXCSDvM8OGC2eFxa_JOEnMnXw8wcYRd-k-uEcNseth1D-STGh2rN5NRUD2X0/s1600/session+commands+sequence.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 6 - Commands run with their sequence</i></td></tr>
</tbody></table>
<br />
<i>Session-1</i> closed 9:57<br />
<i>Session-2</i> closed 9:59<br />
<br />
Session-1 is closed first, followed by Session-2. Here is a snippet of relevant metadata from the resulting files:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJQEW3VJZHDjp0oXWUQcMM7yriVyfvqSGZUB-n20iefv_A2RUNnkAMOCUrBd4-QO6ZUwFhi4q11OEBLPS2LGIMBGs1zZf4ZUhLg6tZUoDmKnTvbI-svxUmJFHmPHpnePypjgoqhY6s_lw/s1600/bashsession_exp_files.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="141" data-original-width="974" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJQEW3VJZHDjp0oXWUQcMM7yriVyfvqSGZUB-n20iefv_A2RUNnkAMOCUrBd4-QO6ZUwFhi4q11OEBLPS2LGIMBGs1zZf4ZUhLg6tZUoDmKnTvbI-svxUmJFHmPHpnePypjgoqhY6s_lw/s1600/bashsession_exp_files.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 7 - Relevant metadata from stat command</i></td></tr>
</tbody></table>
<h3 style="text-align: left;">
Fun Facts</h3>
The start and stop time for a session is available if you look at the crtime (File Created time) for the <i>.history</i> and <i>.historynew</i> files. These are in bold in the screenshot above.<br />
<br />
<span style="background-color: yellow;">Created Time of <i>TERM_SESSION_ID</i>.historynew = Session created time</span><br />
<span style="background-color: yellow;">Created Time of <i>TERM_SESSION_ID</i>.history = Session end time</span><br />
<span style="background-color: yellow;"><br /></span>
<br />
<h3 style="text-align: left;">
Isolating session data</h3>
By comparing the data in various <i>.history</i> files (from different sessions), you can find out exactly which commands belong to a particular session. See pic below, where lines 1-181 (not shown) are from older history (other past sessions). Lines 182-184 are from <i>Session-1</i> and are seen in its history file at the end. <i>Session-2</i> (closed after <i>Session-1</i>) has the same format, ie, old session history with this session's history appended (lines 185-189).<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguVn13ieW7yP_wQndmvdS1SmRGGIP1eCzV9EufvRtVxJ4OTaScShCIuNLXWPFw5YtcEA_CLDb45XGKQCMtEoO0ceaj13qudR1fdiUhZdQqwFXlfBf9jKw-gpH9q_aCny0OiwyUGjH-PLI/s1600/comparing+bashsession+history2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="182" data-original-width="608" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguVn13ieW7yP_wQndmvdS1SmRGGIP1eCzV9EufvRtVxJ4OTaScShCIuNLXWPFw5YtcEA_CLDb45XGKQCMtEoO0ceaj13qudR1fdiUhZdQqwFXlfBf9jKw-gpH9q_aCny0OiwyUGjH-PLI/s1600/comparing+bashsession+history2.png" /></a></td></tr>
<tr><td class="tr-caption" style="font-size: 12.8px;"><i>Figure 8- .history files from Session-1 (Left) and Session-2 (Right)</i></td></tr>
</tbody></table>
<br />
This is easily done in code and the <a href="https://github.com/ydkhatri/mac_apt">mac_apt</a> BASHSESSIONS plugin parses this information to break out the individual commands per session, along with session start and stop time.<br />
<br />
While you still cannot get the exact time when an individual command was run, the sessions functionality does give you a very good narrowed time frame to work with. While we do not have the absolute order of commands ("cp -h" was run before "printenv"), we do have a narrowed time-frame for the set of commands ("cp-h" run between 9:51-9:59 and "printenv" run between 9:44-9:57). This is a big thing for analysts and investigators!<br />
<br />
<br /></div>
Yogesh Khatrihttp://www.blogger.com/profile/18391374024639697695noreply@blogger.com1tag:blogger.com,1999:blog-1264611260322778486.post-14770626926646344892018-02-02T23:07:00.003-05:002018-05-17T16:49:32.056-04:00Reading Notes database on macOS<div dir="ltr" style="text-align: left;" trbidi="on">
The notes app comes built-in with every OSX/macOS release since OSX 10.8 (Mountain Lion). It is great for quick notes and keeps notes synced in the cloud. It can have potentially useful tidbits of user information in an investigation.<br />
<br />
<h3 style="text-align: left;">
Artifact breakdown & Forensics</h3>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Locations </h4>
Depending on version of macOS, this can vary. Sometimes there is more than one database, probably as a result of an upgrade! But only one is actively used at any given time (not both). So far, we haven't seen any duplicate data when two are present.<br />
<br />
<h4 style="text-align: left;">
Location 1</h4>
<blockquote class="tr_bq">
<i><span style="background-color: yellow;">/Users/<USER>/Library/Containers/com.apple.Notes/Data/Library/Notes/</span></i></blockquote>
In here, databases will be named as one of the following:<br />
<blockquote class="tr_bq">
NotesV1.storedata <span style="color: #999999;">← Mountain Lion <i>(Thanks Geoff Black)</i></span><br />
NotesV2.storedata <span style="color: #999999;">← Mavericks</span><br />
NotesV4.storedata <span style="color: #999999;">← Yosemite</span><br />
NotesV6.storedata <span style="color: #999999;">← Elcapitan & Sierra</span><br />
NotesV7.storedata <span style="color: #999999;">← HighSierra</span></blockquote>
If a note has an attachment, then the attachment is usually stored at the following location:<br />
<i><span style="background-color: yellow;">/Users/<USER>/Library/Containers/com.apple.Notes/Data/Library/CoreData/Attachments/UUID/</span></i><br />
<br />
<div>
There does not appear to be much difference between the database types. The following tables have been seen.</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBCNgO0zMuBjDJ1ENQ-B5HeuMGVgrXf5EkAGSDZcqSdre4vPMsEaQPOTNQXQ4XiBg1dI9wHtJR-i2klgKo0Cs4z7BBBc1kzN9ipB18SOvzuDpSBqmB4mpyfx_arrdUfTWMj4giGovrGFI/s1600/NotesV2.tables.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="163" data-original-width="156" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBCNgO0zMuBjDJ1ENQ-B5HeuMGVgrXf5EkAGSDZcqSdre4vPMsEaQPOTNQXQ4XiBg1dI9wHtJR-i2klgKo0Cs4z7BBBc1kzN9ipB18SOvzuDpSBqmB4mpyfx_arrdUfTWMj4giGovrGFI/s200/NotesV2.tables.PNG" width="191" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Tables for NotesV2</i></td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhR6AjbGmGBjZqSx0s0-V-vN_U_f2uT-FWmpIj1wCv0-5vxBcyiQ_IVmLJ1JPWDLecH0SwWD0r-WPizFoEz4cvfXBB5mTLi8NFtwui8ZOI9tsmnjtTPWnKzBolUFVkfYQ_7-wvme745sI4/s1600/NotesV6.tables.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto; text-align: center;"><img border="0" data-original-height="182" data-original-width="151" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhR6AjbGmGBjZqSx0s0-V-vN_U_f2uT-FWmpIj1wCv0-5vxBcyiQ_IVmLJ1JPWDLecH0SwWD0r-WPizFoEz4cvfXBB5mTLi8NFtwui8ZOI9tsmnjtTPWnKzBolUFVkfYQ_7-wvme745sI4/s200/NotesV6.tables.PNG" width="165" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Tables for NotesV6</i></td></tr>
</tbody></table>
<div>
</div>
<div>
<br /></div>
<div>
<div>
Each note can be associated with either a local account or an online one. The following account information can be obtained. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh42PJnKzEgldexj-9ZCzTObkeU89_qFLj_wNWjCrTYo_uBzBtftwIlzggXq0roVCMPDy5RHCrTdMDnnEd5Q6iHT5SQL5A-HN8WOaXujm0Nipv-ESoDzS117MkZIUMmzx9vCCo3SGL-894/s1600/Account.Details.NotesV4.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="104" data-original-width="691" height="60" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh42PJnKzEgldexj-9ZCzTObkeU89_qFLj_wNWjCrTYo_uBzBtftwIlzggXq0roVCMPDy5RHCrTdMDnnEd5Q6iHT5SQL5A-HN8WOaXujm0Nipv-ESoDzS117MkZIUMmzx9vCCo3SGL-894/s400/Account.Details.NotesV4.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="font-size: 12.8px;"><i>Account email address, id and username from ZACCOUNT table</i></td></tr>
</tbody></table>
</div>
<div>
<br /></div>
<div>
Individual note data is stored in ZNOTEBODY and the rest of the tables provide information about note parent folder, sync information, and attachment locations. </div>
<div>
<br /></div>
<div>
Notes converts all data to HTML as seen below.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhD6YXsn8rbSGOjbqNANJYIh-5QBJ3HMPQXjpW3VJuORTc7ef4GDrKseZ_dTkEQRWyvoN4lf8_rOezdWEHd3b-pa6fNwiuzKVS3h4nBrZ9J77EbEh_esBFsVWaxi5s52rcGCCDHUctdDqg/s1600/NotesV4.notedata.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="80" data-original-width="802" height="62" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhD6YXsn8rbSGOjbqNANJYIh-5QBJ3HMPQXjpW3VJuORTc7ef4GDrKseZ_dTkEQRWyvoN4lf8_rOezdWEHd3b-pa6fNwiuzKVS3h4nBrZ9J77EbEh_esBFsVWaxi5s52rcGCCDHUctdDqg/s640/NotesV4.notedata.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>ZNOTES Table with note Html content</i></td></tr>
</tbody></table>
<div>
The graphic below shows how you can find and resolve note attachments to their locations on disk. If an attachment is present, ZNOTEBODY.ZHTMLSTRING will contain the UUID of that which can be matched up to the ZATTACHMENT.ZCONTENTID to get a binary plist blob. When parsed, you can find the full path to the attachment in the plist.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-CGGvLA64Ix2VGGtrfM_uubWnRMsrKBFRp4Pe5hbOvFaG6inh3m_gJaENh5NI0jslzNdbzA__Ahb2YdR33MwCEWFCpn5K9hqCgWWVuNcNBJC2lt9yeoydvxiwpKHq2wr1qBCRmv_XYvY/s1600/NotesV2.attachments.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="343" data-original-width="1037" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-CGGvLA64Ix2VGGtrfM_uubWnRMsrKBFRp4Pe5hbOvFaG6inh3m_gJaENh5NI0jslzNdbzA__Ahb2YdR33MwCEWFCpn5K9hqCgWWVuNcNBJC2lt9yeoydvxiwpKHq2wr1qBCRmv_XYvY/s640/NotesV2.attachments.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Resolving attachment location</i></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
The dates and times fetched are Mac Absolute time, which is the number of seconds since 1/1/2001.<br />
<br />
<h4 style="text-align: left;">
Reading the data</h4>
The following SQL query will pull out most pertinent information from this database:</div>
<div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">SELECT n.Z_PK as note_id, </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">datetime(n.ZDATECREATED + 978307200, 'unixepoch') as created, </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">datetime(n.ZDATEEDITED + 978307200, 'unixepoch') as edited, n.ZTITLE, </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(SELECT ZNAME from ZFOLDER where n.ZFOLDER=ZFOLDER.Z_PK) as Folder,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(SELECT zf2.ZACCOUNT from ZFOLDER as zf1 LEFT JOIN ZFOLDER as zf2 on (zf1.ZPARENT=zf2.Z_PK) where n.ZFOLDER=zf1.Z_PK) as folder_parent,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ac.ZEMAILADDRESS as email, ac.ZACCOUNTDESCRIPTION, </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">b.ZHTMLSTRING, </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">att.ZCONTENTID, att.ZFILEURL</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM ZNOTE as n</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZNOTEBODY as b ON b.ZNOTE = n.Z_PK</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZATTACHMENT as att ON att.ZNOTE = n.Z_PK</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZACCOUNT as ac ON ac.Z_PK = folder_parent</span></blockquote>
<div>
</div>
</div>
<br />
<div>
<h4>
Location 2</h4>
</div>
<div>
<blockquote class="tr_bq">
<span style="background-color: yellow;"><i>/Users/<USER>/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite</i></span></blockquote>
This one has been seen on El Capitan, Sierra and HighSierra. Attachments are stored in the Media folder located here:<br />
<blockquote class="tr_bq">
<i style="background-color: yellow;">/Users/<USER>/Library/Group Containers/group.com.apple.notes/Media/<UUID>/</i></blockquote>
<div>
Here UUID is the unique identifier for each attachment. The database scheme is different here.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><img border="0" data-original-height="326" data-original-width="230" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_OVXyF5Y_lXG_4SwZsRE5QrDHh8Edw1vmbJ-PbiGpahESffUCbvA7RKaY4sm_sZv8PTB9G_OnaW-QozhI52S3cqfkvzd12pOG7iqXaunSxi0dWU2WQ_QOjx5bCbzL5pyBtjR75wn3Gss/s320/NotesStore.ElCapitan.tables.PNG" style="margin-left: auto; margin-right: auto;" width="224" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>NoteStore.sqlite in HighSierra</i></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheLtEnCCdePXPOW9U7KUAewOabdc5MiKSw_Jh3YtzCh5JIW_yr_ppGTIb6ReZBOzHsETfCDcfZptHp-MWBoFM9UoLa0okp4e2MQnSyAJXGsj_An5ihJZ8MVosQBeWB3_ov-ZOpKJmFvrM/s1600/NotesStore.HighSierra.tables.PNG" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" data-original-height="234" data-original-width="228" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheLtEnCCdePXPOW9U7KUAewOabdc5MiKSw_Jh3YtzCh5JIW_yr_ppGTIb6ReZBOzHsETfCDcfZptHp-MWBoFM9UoLa0okp4e2MQnSyAJXGsj_An5ihJZ8MVosQBeWB3_ov-ZOpKJmFvrM/s1600/NotesStore.HighSierra.tables.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>NotesStore.sqlite in ElCapitan</i></td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
Only account name and identifier (UUID) is available. To get full account information, this will need to be correlated with the account info database stored elsewhere.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-e3hhOFmRw_VEvwTkMAXUishoIcn6OvMbsbXYVP6zB88Y6wsAkmc8c5zKdorfQUc6LyFC8dkWiUHQu63heHrlx4P_cLyb7mCtpmjBjNZgLXlQEquC_LHr8PmMxI90miwc2MHsZ0jXUBA/s1600/NotesStore.account.detail.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="110" data-original-width="590" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-e3hhOFmRw_VEvwTkMAXUishoIcn6OvMbsbXYVP6zB88Y6wsAkmc8c5zKdorfQUc6LyFC8dkWiUHQu63heHrlx4P_cLyb7mCtpmjBjNZgLXlQEquC_LHr8PmMxI90miwc2MHsZ0jXUBA/s1600/NotesStore.account.detail.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>ZICLOUDSYNCINGOBJECT account info</i></td></tr>
</tbody></table>
Note data is available in the ZICNOTEDATA table. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEzyRk0EiyM7Ioobw10q8c-cBjRJNRSnY_xqZKa9VmiwULIdypYX71GFyvX5FxOnco_c9yhgP0dXWmEtryBMoQDvBkguVbompzfcfw-I47cXCWlk3blqDGK9f_cs6uWCy_2Zg56gbNCmU/s1600/NotesStore.ZINOTESDATA.table.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="298" data-original-width="529" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEzyRk0EiyM7Ioobw10q8c-cBjRJNRSnY_xqZKa9VmiwULIdypYX71GFyvX5FxOnco_c9yhgP0dXWmEtryBMoQDvBkguVbompzfcfw-I47cXCWlk3blqDGK9f_cs6uWCy_2Zg56gbNCmU/s400/NotesStore.ZINOTESDATA.table.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>ZICNOTEDATA table</i></td></tr>
</tbody></table>
That ZICNOTEDATA.ZDATA blob is gzip compressed. Upon decompression, it reveals the note data stored in a proprietary unknown binary format. As seen below, you can spot the text, its formatting information and attachment info.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh82mb8FjTPumxvJZ0BZfQsPZDENNjlbVAkgEBBWktHXC1wLPQfEqUm-PsSZAPoZ76ab9hxv2Rq3vergQh8Cbyeki-jRdFWs5x-QP-tYuJAuR_bXmb3Ou3IeQfheItqbqwZr4QXkxuNvN8/s1600/NotesStore.Notedata.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="70" data-original-width="572" height="76" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh82mb8FjTPumxvJZ0BZfQsPZDENNjlbVAkgEBBWktHXC1wLPQfEqUm-PsSZAPoZ76ab9hxv2Rq3vergQh8Cbyeki-jRdFWs5x-QP-tYuJAuR_bXmb3Ou3IeQfheItqbqwZr4QXkxuNvN8/s640/NotesStore.Notedata.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>ZDATA gzipped blob showing gzip signature (1F8B08)</i></td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzMCME468cgxZDqQouOX_eHEmmbCfYi3Gls98KpK6rIkVhyphenhyphenm78fdJl9OvGRtKW6v5kX-WqExkz288xhLhGkNPutZynHTuPglbTTZ0nZ9nLGq3J7xq2DRLYUb6e3yR9BMWoOZzH8vTQMBg/s1600/NotesStore.Notedata.decompressed.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="374" data-original-width="711" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzMCME468cgxZDqQouOX_eHEmmbCfYi3Gls98KpK6rIkVhyphenhyphenm78fdJl9OvGRtKW6v5kX-WqExkz288xhLhGkNPutZynHTuPglbTTZ0nZ9nLGq3J7xq2DRLYUb6e3yR9BMWoOZzH8vTQMBg/s640/NotesStore.Notedata.decompressed.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Uncompressed ZDATA blob showing text, formatting and attachment info</i></td></tr>
</tbody></table>
<div>
<br />
Most notable in this database is the presence of several timestamps-</div>
<blockquote class="tr_bq">
<i><span style="color: blue;">Note Title Modified → ZDATEFORLASTTITLEMODIFICATION</span></i><br />
<i><span style="color: blue;">Note Created → ZCREATIONDATE</span></i><br />
<i><span style="color: blue;">Note Modified → ZMODIFICATIONDATE1</span></i><br />
<i><span style="color: #660000;">Attachment Modified → ZMODIFICATIONDATE</span></i><br />
<i><span style="color: #660000;">Attachment Preview Updated → ZPREVIEWUPDATEDATE</span></i></blockquote>
<div>
<h4 style="text-align: left;">
Reading NoteStore.sqlite</h4>
<div>
<br />
The following SQL query will pull out most pertinent information from this database:</div>
<div>
<blockquote>
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">SELECT n.Z_12FOLDERS as folder_id , n.Z_9NOTES as note_id, d.ZDATA as data,<br />c2.ZTITLE2 as folder,<br />datetime(c2.ZDATEFORLASTTITLEMODIFICATION + 978307200, 'unixepoch') as folder_title_modified,<br />datetime(c1.ZCREATIONDATE + 978307200, 'unixepoch') as created,<br />datetime(c1.ZMODIFICATIONDATE1 + 978307200, 'unixepoch') as modified,<br />c1.ZSNIPPET as snippet, c1.ZTITLE1 as title, c1.ZACCOUNT2 as acc_id,<br />c5.ZACCOUNTTYPE as acc_type, c5.ZIDENTIFIER as acc_identifier, c5.ZNAME as acc_name,<br />c3.ZMEDIA as media_id, c3.ZFILESIZE as att_filesize,<br />datetime(c3.ZMODIFICATIONDATE + 978307200, 'unixepoch') as att_modified,<br />datetime(c3.ZPREVIEWUPDATEDATE + 978307200, 'unixepoch') as att_previewed,<br />c3.ZTITLE as att_title, c3.ZTYPEUTI, c3.ZIDENTIFIER as att_uuid,<br />c4.ZFILENAME, c4.ZIDENTIFIER as media_uuid<br />FROM Z_12NOTES as n<br />LEFT JOIN ZICNOTEDATA as d ON d.ZNOTE = n.Z_9NOTES<br />LEFT JOIN ZICCLOUDSYNCINGOBJECT as c1 ON c1.Z_PK = n.Z_9NOTES<br />LEFT JOIN ZICCLOUDSYNCINGOBJECT as c2 ON c2.Z_PK = n.Z_12FOLDERS<br />LEFT JOIN ZICCLOUDSYNCINGOBJECT as c3 ON c3.ZNOTE = n.Z_9NOTES<br />LEFT JOIN ZICCLOUDSYNCINGOBJECT as c4 ON c3.ZMEDIA = c4.Z_PK<br />LEFT JOIN ZICCLOUDSYNCINGOBJECT as c5 ON c5.Z_PK = c1.ZACCOUNT2<br />ORDER BY note_id</span> </span></blockquote>
On HighSierra, use this query:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">SELECT n.Z_PK, n.ZNOTE as note_id, n.ZDATA as data,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">c3.ZFILESIZE,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">c4.ZFILENAME, c4.ZIDENTIFIER as att_uuid,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">c1.ZTITLE1 as title, c1.ZSNIPPET as snippet, c1.ZIDENTIFIER as noteID,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">datetime(c1.ZCREATIONDATE1, 'unixepoch') as created, </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">datetime(c1.ZLASTVIEWEDMODIFICATIONDATE, 'unixepoch'), </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">datetime(c1.ZMODIFICATIONDATE1, 'unixepoch') as modified,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">c2.ZACCOUNT3, c2.ZTITLE2 as folderName, c2.ZIDENTIFIER as folderID,</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">c5.ZNAME as acc_name, c5.ZIDENTIFIER as acc_identifier, c5.ZACCOUNTTYPE</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM ZICNOTEDATA as n</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZICCLOUDSYNCINGOBJECT as c1 ON c1.ZNOTEDATA = n.Z_PK</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZICCLOUDSYNCINGOBJECT as c2 ON c2.Z_PK = c1.ZFOLDER</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZICCLOUDSYNCINGOBJECT as c3 ON c3.ZNOTE= n.ZNOTE</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZICCLOUDSYNCINGOBJECT as c4 ON c4.ZATTACHMENT1= c3.Z_PK</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">LEFT JOIN ZICCLOUDSYNCINGOBJECT as c5 ON c5.Z_PK = c1.ZACCOUNT2</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ORDER BY note_id</span></blockquote>
If you are looking for an automated way to read this, use <a href="https://github.com/ydkhatri/mac_apt" target="_blank">mac_apt</a>, the NOTES plugin will parse it.<br />
<br /></div>
</div>
</div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/03726664886311447808noreply@blogger.com18tag:blogger.com,1999:blog-1264611260322778486.post-7731023071835412912017-12-12T23:09:00.002-05:002017-12-12T23:16:00.510-05:00mac_apt + APFS support<div dir="ltr" style="text-align: left;" trbidi="on">
Over the past few months, I've been working at adding APFS support into mac_apt, and its finally here. Version 0.2 of <a href="https://github.com/ydkhatri/mac_apt" target="_blank">mac_apt</a> is now available with APFS support. It also adds a new plugin to process print jobs, some enhanced functionality in other plugins and several minor bug fixes.<br />
<br />
As of now basic APFS support is complete, mac_apt can view and extract any file on the file system, including compressed files. It does not have support for FileVault2 (encryption) and will not handle an encrypted volume. The checkpoint feature in APFS is currently not supported or tested although this may be added later.<br />
<br />
This is the first forensic processing tool (in freeware) to support APFS. I believe at this time, Sumuri Recon is the only commercial one. I am unaware of any other that can read APFS.<br />
<br />
I would like to thank Kurt-Helge Hansen for publishing the paper detailing APFS internal structure and working. He was also helpful in providing a proof of concept code for the same.<br />
<br />
The implementation we've used is based on the APFS template built with <a href="http://kaitai.io/" target="_blank">kaitai-struct</a>. Kaitai-Struct is a library that makes it easy to define and read C structures. It will generate all the code required to read those structures. For APFS, the kaitai-struct template was developed originally by Jonas Plum (@cugu_pio) & Thomas Tempelmann (@tempelorg) <a href="https://github.com/cugu/apfs.ksy" target="_blank">here</a>.<br />
<br />
<h3 style="text-align: left;">
APFS working and implementation</h3>
The approach we've taken is to read all inodes and populate a database with this data. This means we have to read the entire filesystem data upfront before we have information to read a single file. It isn't ideal, it practically takes 2-4 min. time to do this on an image having default macOS installation (using my slow regular SATA III external disk over USB3), which is not too bad. But I opted for this path as it is the only solution available for now. Why? The way APFS stores files in its b-tree, they are not sorted by name alphabetically. Instead, a 3 byte hash is computed for each file name and the b-tree maintains nodes sorted by this hash instead. The problem is that this hash algorithm is currently unknown. It may just be some sort of CRC variant or something very different. Until this algorithm is known, we cannot write a native parser that walks the b-tree. Hence the database for now.<br />
<br />
The database does offer us several advantages though. For compressed file information, we can pre-process the logical size and save that for quick retrieval. In APFS, a compressed file will have its logical size set to zero in file metadata. To lookup its real size, you have to go read its compressed data header (which may be inline or in a resource fork), parse it and get the uncompressed (logical) size. This often means going out to an extent to read it, which makes it slow. Pre-populating this info in a database makes it much quicker for later analysis.<br />
<div>
<br /></div>
APFS allows extended attributes to be defined and used just the same as HFS+. This means a file can have extended attributes and those are used to save compression parameters (similar to HFS+). APFS also uses Copy-On-Write, which means if you copy a file, the resulting copy will not duplicate the data on disk. Both inodes (original and copy) will point to the same original extents. Only when the copy is changed will new extents be allocated.<br />
<br />
If you are not familiar with APFS, the Disk Info output from mac_apt might look strange to you.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyMhLSrtMhw41tKpvK728OSBR7FNFu0Rtk6xmGyHBESTCs_XJu8LUApa95LRiR4Z1Og0yAptcnFMo6hFh6d7A-qCslNmTT1QBaSjRuLBRcUObO1FgM_koagiwZ6kRPqS26D-dxgb5LJ1c/s1600/APFS-partitions.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="141" data-original-width="740" height="121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyMhLSrtMhw41tKpvK728OSBR7FNFu0Rtk6xmGyHBESTCs_XJu8LUApa95LRiR4Z1Og0yAptcnFMo6hFh6d7A-qCslNmTT1QBaSjRuLBRcUObO1FgM_koagiwZ6kRPqS26D-dxgb5LJ1c/s640/APFS-partitions.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Screenshot - Disk Info data from mac_apt showing same offset & size for all APFS volumes</i></td></tr>
</tbody></table>
This may be read as 4 partitions all type APFS having the same exact starting offset and size! The reason for this is that APFS is a little different. It isn't just defining a volume, rather it implements a container which can host several volumes in it! This output is from a default installation of HighSierra, where the Disk partitioning scheme is GPT and it defines 2 partitions as seen in the screenshot below.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXaXbtnNMsAXUeQ8kcAtOB_ufRkvND_9fp0x1jmVfedSrMhPyXatgbuaKOoBjQXk-6t3u5b0avxSNZN54Dg-h9cKQqxDLVD0_kKlJjI__hLPuufykYMeYcB8NRhqRaEuiA0p59qyEbN3g/s1600/apfs-gpt.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="381" data-original-width="540" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXaXbtnNMsAXUeQ8kcAtOB_ufRkvND_9fp0x1jmVfedSrMhPyXatgbuaKOoBjQXk-6t3u5b0avxSNZN54Dg-h9cKQqxDLVD0_kKlJjI__hLPuufykYMeYcB8NRhqRaEuiA0p59qyEbN3g/s320/apfs-gpt.PNG" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Illustration showing APFS container and volumes within the Disk</i></td></tr>
</tbody></table>
<br />
The APFS container by default does not put a limit on the size or location of those volumes within it (Preboot, Macintosh HD, VM, Recovery). Unlike normal partitions on disk where sectors are allocated for each volume before you can use the volumes, APFS allows all volumes to share a common pool of extents (clusters) and they all report having total free space as the same. But really its shared space, so you cannot sum it up for all volumes. This also means data from all volumes is interspersed and volumes are not contiguous. The design makes sense as the target media is flash memory (SSD), and it will never be contiguous there (as it did on spinning HDDs) because of the way flash memory chips work.<br />
<br /></div>
Yogesh Khatrihttp://www.blogger.com/profile/03726664886311447808noreply@blogger.com0tag:blogger.com,1999:blog-1264611260322778486.post-76589882909836508742017-09-28T21:29:00.000-04:002018-09-18T11:09:09.323-04:00APFS timestamps<div dir="ltr" style="text-align: left;" trbidi="on">
From <a href="https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/APFS_Guide/VolumeFormatComparison/VolumeFormatComparison.html#//apple_ref/doc/uid/TP40016999-CH8-DontLinkElementID_22" target="_blank">APFS documentation</a> it is revealed that the new timestamp has a nano second resolution. From my data I can see several timestamps in what appears to be the root directory for a test APFS volume I created. Armed with this information, it was pretty easy to guess the epoch - it is the Unix epoch.<br />
<div>
<br /></div>
<blockquote class="tr_bq">
<b><i>ApfsTimeStamp = number of nano-seconds since 1-1-1970</i></b></blockquote>
<div>
<br /></div>
<div>
If you like to use the 010 editor for analysis, put the following code in your Inspector.bt or InspectorDates.bt file:</div>
<div>
<span style="font-size: x-small;"><br /></span></div>
<div>
<div>
<span style="color: lime; font-family: "courier new" , "courier" , monospace; font-size: x-small;">//----------------------------------------------------------------</span></div>
<div>
<span style="color: lime; font-family: "courier new" , "courier" , monospace; font-size: x-small;">// ApfsTime</span></div>
<div>
<span style="color: lime; font-family: "courier new" , "courier" , monospace; font-size: x-small;">// 64-bit integer, number of nanoseconds since 01/01/1970 00:00:00</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: blue;">typedef</span> int64 ApfsTime <read=ApfsTimeRead, write=ApfsTimeWrite>;</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FSeek( pos ); ApfsTime _aft <name="ApfsTimes">;</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">string ApfsTimeRead( ApfsTime t )</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">{ </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // Convert to FILETIME</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: blue;">return</span> FileTimeToString( t/100L + 116444736000000000L );</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">int ApfsTimeWrite( ApfsTime &t, string value )</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">{</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // Convert from FILETIME</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FILETIME ft;</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> int result = StringToFileTime( value, ft );</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> t = (((int64)ft - 116444736000000000L)*100L);</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: blue;">return</span> result;</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span></div>
</div>
<div>
<br />
Now 010 can interpret APFS timestamps :)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivwrSaov_TMefoJw0qsrzIZQylsVZ400B_l5saqErDl2OiHMEuyQzHIxUrdkBJbRaGeRTMEfjX_kq3b-fIDgxoxP83Btk28U0ESO3eGEeyEP4sMjWhbp5cduIRju5X9_avlbtDRbzJF2c/s1600/APFStimestamps.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="105" data-original-width="582" height="70" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivwrSaov_TMefoJw0qsrzIZQylsVZ400B_l5saqErDl2OiHMEuyQzHIxUrdkBJbRaGeRTMEfjX_kq3b-fIDgxoxP83Btk28U0ESO3eGEeyEP4sMjWhbp5cduIRju5X9_avlbtDRbzJF2c/s400/APFStimestamps.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>010 interprets ApfsTimes</i></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /></div>
If you need to read this timestamp in python (and convert to python datetime), the following function will do this:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br />import <span style="color: #674ea7;">datetime</span></span><br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br />def <span style="color: purple;"><b>ConvertApfsTime</b></span>(ts):<br /> try:<br /> return datetime.datetime(1970,1,1) + datetime.timedelta(microseconds=ts / 1000. )<br /> except:<br /> pass<br /> return <span style="color: purple;">None</span></span></div>
<div>
<br />
<i><u>Update</u>: As noted by Joachim, the timestamp is an int64, and has been corrected in the above code.</i></div>
</div>
Yogesh Khatrihttp://www.blogger.com/profile/03726664886311447808noreply@blogger.com3tag:blogger.com,1999:blog-1264611260322778486.post-53960607639325860152017-09-18T16:28:00.001-04:002017-12-12T20:43:09.368-05:00Interpreting volume info from FXDesktopVolumePositions<div dir="ltr" style="text-align: left;" trbidi="on">
On a mac, if you wanted to get a list of all connected volumes, you would typically lookup FXDesktopVolumePositions under the <i>com.apple.finder.plist</i> file (for each user). This is available under <i>/Users/<USER>/Library/Preferences/</i> The volumes listed here can include anything mounted, a CD, a DMG file, a volume from external disk, network mapped volume or anything else that mac sees as a mounted volume/device.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLxlpRIk_U-rak3e3s_C1uH20909fP6xMP9izstmRqn1XL_ZgDwEnNGnj03Xt5MPAHcfNLXnUr6MEtYh_XutyV81y1RNoOID3sm6L-TX-0KCmbaZLfsEO4hgPLdkh1m6Je02WBRB3mOtQ/s1600/fxvolumedesktoppositions_snippet.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="162" data-original-width="401" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLxlpRIk_U-rak3e3s_C1uH20909fP6xMP9izstmRqn1XL_ZgDwEnNGnj03Xt5MPAHcfNLXnUr6MEtYh_XutyV81y1RNoOID3sm6L-TX-0KCmbaZLfsEO4hgPLdkh1m6Je02WBRB3mOtQ/s1600/fxvolumedesktoppositions_snippet.PNG" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 1: Snippet of 'FKDesktopVolumePositions' from com.apple.finder.plist</i></td></tr>
</tbody></table>
<br />
The above snippet shows every volume connected to one of our test machines. What's strange is the way the volume information is stored. Ignoring the icon position data (which resides a level below not shown in that screenshot), all that's available is a string that looks like this:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<div style="text-align: center;">
<span style="font-family: "courier new" , "courier" , monospace;"><b>HandBrake-0.10.2-MacOSX.6_GUI_x86_64_0x1.b2a122dp+28</b></span></div>
<br />
Studying this a bit, it appears that the random looking hex number after the name is not really random. It represents the '<i>Created date</i>' of the volume's root folder (for most but not all the entries!). The date format is Mac Absolute Time (ie, number of seconds since 2001).<br />
The picture below shows the breakup.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYBXXz6R6HT1hpcofPdHwjoaXEptJmuzK6kK_riBckG-RJLaHKXQpC1DfphC2BLaRpzVtPrXQ_bEVugxBxsgUcpayAagLEIPfsRZO3uiqm-xKaWZBtZ2rqDfNIIQMe5ClcHxuE9PRmxUw/s1600/fxvolumedesktoppositions.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="333" data-original-width="770" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYBXXz6R6HT1hpcofPdHwjoaXEptJmuzK6kK_riBckG-RJLaHKXQpC1DfphC2BLaRpzVtPrXQ_bEVugxBxsgUcpayAagLEIPfsRZO3uiqm-xKaWZBtZ2rqDfNIIQMe5ClcHxuE9PRmxUw/s400/fxvolumedesktoppositions.PNG" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Figure 2: Interpreting volume info from FXDesktopVolumePositions entry</i></td></tr>
</tbody></table>
<br />
<br />
The first part of the entry (till the last underscore) is the name as it appears in the Finder window. The next part ignoring the period is the created date. <strike>This is followed by <b><span style="color: #3d85c6; font-family: "courier new" , "courier" , monospace;">p+</span></b> and then a decimal number (usually 0, 26, 27, 28 or 29). I believe that number represents a volume type. From what I can see in some limited testing, if volume type=28, then the hex number is always the created date.</strike><br />
<strike><br /></strike>
<strike>Large external disks (USB) that identify as fixed disks and most DMG volumes will show up as type 28. All USB removable disks show up under type 29 and here the hex number is not a created date, it is unknown what this may be. Sometimes this number is negative for type 29, and a lot of volumes share the same number.</strike><br />
<strike><br /></strike>
<strike>There is still some mystery about the hex number. For type=28, sometimes it is less than 8 digits long, and needs to be padded with zeroes at the end. This does produce an accurate date then. Also, sometimes it is longer than 8 digits! In these cases, truncating the number at 8 digits has again produced an accurate date in limited testing. It is unclear what those extra digits would denote.</strike><br />
<br />
<u>Update</u>: As pointed out by Geoff Black, the part after the underscore is just the way floats are represented (including the <b><span style="color: #3d85c6; font-family: "courier new" , "courier" , monospace;">p+</span></b>xx part).<br />
<br />
<i>Following this discovery, <a href="https://github.com/ydkhatri/mac_apt" target="_blank">mac_apt</a> has been updated to parse this date.</i></div>
Yogesh Khatrihttp://www.blogger.com/profile/03726664886311447808noreply@blogger.com4