[How-To] Using GnuPG to verify data using detached signatures

Many software downloads come with a signature file. You normally need to download this signature file separately. Signatures are a great way to let people know that you are the person / company that is making the software available, and that no one else has changed the data since its release.

Fig 1: Tails ISO and signature file download

We are going to use Tails Linux as an example. On their download page, you will find a link to download the Tails ISO image. This is the data we are interested in running. Think of it like the main program that we want to install / use.

Next, we are given a link to the "Tails 1.4 signature". This is signature file that the distributor created. With this signature we can verify that the Tails ISO Image has not been modified by anyone else.

Tails also provides a "SHA256 Checksum". This is a less-rigorous way than signatures to verify the data has not changed.

First, download the ISO file AND the signature file. The signature file will almost always end with ".sig". Make sure both files are in the same directory.

Once you had both files, open the command line / terminal and navigate to that directory. Next we need to use gpg to verify the signature. If we try to verify now, we may get the following results:

gpg2 --verify tails-i386-1.4.iso.sig gpg: assuming signed data in 'tails-i386-1.4.iso'
gpg: Signature made Tue 12 May 2015 02:56:27 AM KST using RSA key ID 752A3DB6
gpg: Can't check signature: No public key

In this case, we also need to get the public key of the person that created the signature. From the tails website, I find the ID of their signing key, so now we need to import.

gpg2 --recv-keys A490D0F4D311A4153E2BB7CADBB802B258ACD84F
gpg: key 58ACD84F: public key "Tails developers (offline long-term identity key) " imported
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: next trustdb check due at 2017-01-09
gpg: Total number processed: 1
gpg:               imported: 1

Make sure we have the right key:

gpg2 --list-keys
pub   rsa4096/58ACD84F 2015-01-18 [expires: 2016-01-11]
uid       [ unknown] Tails developers (offline long-term identity key) 
sub   rsa4096/752A3DB6 2015-01-18 [expires: 2016-01-11]
sub   rsa4096/2F699C56 2015-01-18 [expires: 2016-01-11]

Now verify the signature again:

gpg2 --verify tails-i386-1.4.iso.sig gpg: assuming signed data in 'tails-i386-1.4.iso'
gpg: Signature made Tue 12 May 2015 02:56:27 AM KST using RSA key ID 752A3DB6
gpg: Good signature from "Tails developers (offline long-term identity key) " [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A490 D0F4 D311 A415 3E2B  B7CA DBB8 02B2 58AC D84F
     Subkey fingerprint: BA2C 222F 44AC 00ED 9899  3893 98FE C6BC 752A 3DB6

Here we can see when the signature was made, and the ID of that key. Next we see "Good signature" which means that the signature does verify the data.

Remember, we were given the SHA256 value of the ISO file. Get the SHA256 hash with the following command (linux):

sha256sum tails-i386-1.4.iso
339c8712768c831e59c4b1523002b83ccb98a4fe62f6a221fee3a15e779ca65d  tails-i386-1.4.iso

Now can can compare this hash value to the one on the website, and we see that they are the same.

If I can just check the hash value, why verify with a signature?

Hash values do allow you to make sure that the data has not changed, however, there are a number of weaknesses. For example, someone intercepting your network traffic could deliver the web page to you with an altered ISO link AND an altered hash value on the page. This means that the hash value will be valid, but the source of the information cannot be trusted.

Signatures help this in a number of ways. Because the signature is generated by a developer's private key, and we are verifying it with their public key, it is nearly impossible for someone to pretend to be the developer. Also, since we did not download the public key from the webpage, but looked it up on a different server, it is slightly more difficult for someone to trick us into download the wrong key. Further, we can try to use the Web of Trust to make sure we are getting the right key. In our case, we can see who has signed this key by going to a keyserver checking.

Read More

Clearing USB disk read cache for testing and forensics in Linux

When copying data from USB devices in Linux (Debian / Ubuntu), you may have noticed that reading data from the disk the first time takes a while, and reading the second time takes only a few seconds.

For example:

joshua@Apollo /media/joshua/ucdntfs $ time sudo md5sum /dev/sdf1
3a698f0c3155e494274e5e7829f4d246  /dev/sdf1

real 2m58.620s
user 0m7.032s
sys 0m1.429s

joshua@Apollo /media/joshua/ucdntfs $ time sudo md5sum /dev/sdf1
3a698f0c3155e494274e5e7829f4d246  /dev/sdf1

real 0m3.467s
user 0m3.285s
sys 0m0.181s

Here the first read took 2 minutes 58 seconds, while the second took only 3 seconds.This is because all data on the disk is cached to memory when read the first time. In cases where the disk may change between reads, caching may return results that are not consistent with the current state of the disk (like a hash).

When looking how to disable read cache, I found a lot of information about disabling write cache, but not a lot about disabling read.

To disable write cache (if supported) for the current session that the device is plugged in:

sudo hdparm -W 0 /dev/[device]

But this does not solve our read cache problems. Unfortunately, I could not find a way to completely disable read cache, but we can clear the cache buffer.

First, determine the path to echo with

which echo

Then we want to tell the kernel to drop caches. To do this, we need to echo a value to /proc/sys/vm/drop_caches.

To free pagecache:
 echo 1 > /proc/sys/vm/drop_caches
To free reclaimable slab objects (includes dentries and inodes):
 echo 2 > /proc/sys/vm/drop_caches
To free slab objects and pagecache:
 echo 3 > /proc/sys/vm/drop_caches

So our echo command to clear all caches should look like:

sudo sh -c "/bin/echo 3 > /proc/sys/vm/drop_caches"

Note: You probably cannot echo directly to drop_caches with sudo - you should be root. The work-around to that is wrap the whole command in sudo. Make sure you are putting the full path to echo on your system.

joshua@Apollo /media/joshua/ucdntfs $ time sudo md5sum /dev/sdf1
3a698f0c3155e494274e5e7829f4d246  /dev/sdf1

real 3m18.294s
user 0m6.389s
sys 0m1.390s

joshua@Apollo /media/joshua/ucdntfs $ sudo sh -c "/bin/echo 3 > /proc/sys/vm/drop_caches"

joshua@Apollo /media/joshua/ucdntfs $ time sudo md5sum /dev/sdf1
3a698f0c3155e494274e5e7829f4d246  /dev/sdf1

real 3m18.344s
user 0m6.545s
sys 0m1.438s

If you use it a lot, like me, you might want to make an alias:

alias clearusbcache="sudo sh -c '/bin/echo 3 > /proc/sys/vm/drop_caches'"

If you want to clear the cache in the background while running experiments, you can try this script:

#!/bin/bash
while true; do
   /bin/echo 3 > /proc/sys/vm/drop_caches
   sleep 1
done

Read More

Convert EnCase hash sets to md5sum

I managed to get a hold of a list of known-bad hashes to use in an experiment. The hashes, however, were in EnCase ".hash" format.

I am mostly using the SleuthKit's hfind to do some hash comparisons. My setup could already use the NSRL hash sets with no problem, and TSK is supposed to support EnCase hash sets. I was able to create an index for the EnCase hash sets, but when I attempted to query, I would get an error:

Command: hfind db.hash [hash value] 
Error: "Cannot determine hash database type (hdb_setupindx: Unknown 
Database Type in index header: encase)

No responses when asking about the error on the mailing list, so I looked for other ways to access the hashes.

Finally, I came across Jesse Kornblum's EnCase hash file converter (encase2txt). The tool built fine in Linux (Ubuntu), and the Windows binary worked with no issue on Windows 7 (64bit).

Just point the tool at the EnCase hash database and it will output all the hashes in a format like md5sum. Pipe this plain text output to a file, and you have an md5sum hash file. From this I was able to build the index (hfind -i md5sum hashes.md5) and query the database with no problems.

Thanks Jesse!

Building and usage: http://jessekornblum.livejournal.com/166275.html

Read More

Using XARGS to speed up batch processing

[Edit 24/7/2013] Be careful when using xargs to spawn multiple processes that write to the same file. I've been using it with md5sum and piping the output to a file. In data sets with a large number of files, several hundred hashes would be missing in the final file. A more reliable way is to use md5deep.


In our last post we compared md5sum, md5deep and openssl md5 to determine which is fastest at calculating different hashes. For that experiment we were using 1 thread. Essentially, a file comes in, it gets hashes, the next file comes in. With 1 threat, we saw 1 out of the 4 processors being used.

However, if we are processing a list of files, why not split the work into multiple jobs and use all of the processors. md5deep does this by default (4 threads), but splitting the jobs for any task is easily done with xargs.

xargs is a program that allows you to control arguments coming in from a pipe. For example, if we type 'find -type f | md5sum' then the list of found files would be hashed as one glob of data. To give each file name to md5sum, instead of all file names, we can use xargs to control how the output of find is piped to md5sum. For example, 'find -type f | xargs -n1 md5sum' will feed one line at a time to md5sum, allowing md5sum to find the file and hash it.

You can also tell xargs how many threads to create using the -P switch. Since we have 4 processors, we will use 4 threads in this case.

Consider in the last post, were the times for hashing all files in a directory were as follows:
md5sum: real 3m46.117s
md5deep: real 3m57.595s
openssl: real 3m43.142s

(example: 'find -type f | xargs -P4 -n1 md5sum')
Using xargs (and all of the processors), the real times can be reduced to:
md5sum: real 1m57.196s
md5deep: real 2m5.408s
openssl: real 1m56.084s

md5deep recursive, default threads (4), no xargs:
real 2m0.521s

Note: the user and system time is the same as before because the same amount of processor time is being devoted to the tasks, however, there are more processors so the real time is much less.

Also, consider that all the files in the directory are different sizes, with the largest being 10GB. The majority of the time for each program was spent with 1 thread hashing the largest file, while the other processors had already finished hashing all the other files.

Read More

md5sum vs. md5deep vs. openssl md5: MD5 calculation speed test

The following experiment is conducted to determine if md5sum, md5deep or openssl md5 hash calculations are faster than the others.

Methodology:

Test 1: A directory of test files consisting of disk images and extracted Windows Registry files will be scanned with the command 'find -type f | xargs -n 1 md5program'. The output of this command will be fed into each program, respectively. The time for the entire process to finish will be tracked with the time command. Xargs is being used to feed each line of output to the md5 program.

Test 1.1: After all programs have finished, the computer will be restarted, and the order the programs are ran in will be changed. For example, if md5sum was the first program in the first round, it will be the second program in the second round. This is to attempt to account for any caching that takes place in the operating system.

The result will be 9 timed runs for calculating the md5 sum of all files in a directory.

Test 2: Big files - In test 1, multiple files of different sizes will be continuously fed to the hashing program. In test 2, a single 10 GB disk image will be given to each program and timed.

Test machine:

Intel(R) Core(TM) i7-3537U CPU @ 2.00GHz (4 cores)

8 GB RAM

Timing cached reads: 8892.40 MB/sec

Timing buffered disk reads: 403.98 MB/sec

Results:

md5sum, run 1	md5sum, run 2	md5sum, run 3
real 3m46.457s	real 3m46.117s	real 3m59.198s
user 0m42.183s	user 0m39.254s	user 0m42.235s
sys 3m3.595s	sys 3m6.300s	sys 3m16.096s
md5deep, run 1	md5deep, run 2	md5deep, run 3
real 3m57.595s	real 3m57.666s	real 4m2.255s
user 0m43.743s	user 0m43.567s	user 0m44.551s
sys 3m17.552s	sys 3m17.192s	sys 3m22.881s
openssl md5, run 1	openssl md5, run 2	openssl md5, run 3
real 3m48.130s	real 3m43.142s	real 3m50.619s
user 0m37.878s	user 0m37.462s	user 0m38.558s
sys 3m9.436s	sys 3m5.112s	sys 3m11.136s

Overall processor time:
Run 1, md5sum 3m45.77s; md5deep 4m01.29s; openssl 3m47.32s.
Run 2: md5sum 3m45.55s; md5deep 4m00.76s; openssl 3m42.57s.
Run 3: md5sum 3m58.34s; md5deep 4m07.43s; openssl 3m49.7s.

Average (rounded to second):
md5sum: 3m50s
md5deep: 4m03s
openssl: 3m46s

Test 1 Conclusions
While the 'real' time may be an interesting factor for investigators, what we are more interested in is the time that the program was using the processor. In this case 'user' is the time taken by the program in user mode, and sys is time in kernel mode. The basic idea is that we want a program that takes the least amount of time on the processor to do the same amount of work.

Note: from the data it appears that another process, perhaps an OS update, was taking place that affected all the programs.

Based on the average of the three runs, it appears that openssl is slightly faster, followed by md5sum and then md5deep.

It should be noted, however, that md5deep is not really being used in the way it was designed. For example, md5sum does not include a recursive mode, where md5deep does. If we run md5deep using its recursive mode - instead of find and xargs - then the time is actually slightly better than the others:

md5deep recursive - restricted to 1 thread
real 3m43.963s
user 0m41.327s
sys 3m2.103s

md5deep: 3m43s

So, in conclusion, if you are are feeding a list of files into an md5 hash program, openssl appears to be a slightly better choice over md5sum and md5deep. However, if you can choose how the file list is ingested, md5deep is probably a better choice because of speed and available features.

Test 2: Testing the time for each program to hash a 10 GB disk image. This test will not use find and xargs - it will use the program directly. md5deep will be restricted to 1 thread.

md5sum	md5deep	openssl md5
real 1m44.558s	real 1m49.491s	real 1m43.475s
user 0m19.369s	user 0m20.421s	user 0m18.233s
sys 1m24.877s	sys 1m30.534s	sys 1m24.889s

md5sum: 1m44s
md5deep: 1m51s
openssl: 1m43s

For a single large file, it appears that openssl is also the fastest, followed by md5sum and then md5deep.

Please note: all of these tests are, at best, quite shallow. A proper testing environment with many more runs should be conducted.

---

Bonus observation: As set, xargs will send 1 line to the hashing program and spawn 1 process. This process seems to switch between processors, even while the same hash is calculating.

Read More