multiSum

multiSum_v1.4.1 ReadMe

MultiSum is a command line file hasher. Most of the other tools I've seen only focus on a single hash, or don't allow for parallel calculations. Some of my applications needed to collect the CRC, md5 and sha from a bunch of large files, which is painful if you have to do three reads. This program makes the most common hashes available in one tool and can calculate them all on a single pass through a given file; it uses one thread per selected hash to leverage multi-core CPU's. An interesting side effect is that if you run enough hashes in parallel you will be restricted by memory bandwidth, rather than CPU or IO.

In addition to computing many hashes in parallel, this tool also features Unicode and long path support, and can handle NTFS alternate data streams.

Output is completely customizable, so you can use multiSum to generate .sfv files, .md5 files, or anything else. Check the examples in the usage instructions. Supported hashes are: Fletcher32, Adler32, CRC32 (optimized implementation, outperforms the native windows function), MD2, MD4, MD5, RIPEMD160, SHA1, various SHA2, various SHA3, various Tiger, various Blake2B, and Whirlpool. KangarooTwelve & others might be coming in a future version.

The tool also has a verification mode for processing existing checksum files; the following types are supported: .sfv, .md5, .sha1, .sha256, .blake2/.blake2b.

Finally, a benchmarking mode is included to test hash speeds.

Note on CNG versions: Cryptograny Next Generation (CNG) versions of supported hashes are made available. These have identical output to their non-CNG counterparts but can be faster to calculate due to hardware acceleration. The SHA256CNG and MD4CNG versions in particular are quite a lot faster for me, while MD2 still sucks; run the benchmark mode to see if they'll help. Thanks to Andrew Arnott for collecting the bcrypt.dll p/invokes. (These may not work on very old windows versions, as CNG was introduced in Windows vista, so I'm keeping the old managed code versions of MD2 and MD4 as they are, they don't take up much space anyway.)

Note on tiger: The TIGER4_ variants are an extended round version (8 extra rounds) with output identical to php's "tiger192,4". I have no idea what the impetuous of it was, or if there's a settled standard for the extended round version, but whatever, it seems to be popular...

Usage Notes

Version 4.6.2 or better of the .net framework is recommended. Get it from Microsoft.

No installation; just unpack and run.

Command-line Usage:

multiSum.exe [-s:SUBDIRECTORY_FLAG] [-o:OUTPUT_FORMAT] FILES_AND_FOLDERS
   File or directory names must be quoted if they contain spaces.
   More than one file or folder can be listed, separated by spaces.

SUBDIRECTORY_FLAG:
   Must be either on (-s:1) or off (-s:0)
   If any directories are specified, this flag determines if we hash files found in all subdirectories or only those immediately inside the specified directory.
   Defaults is no subdirectory processing: -s:0

OUTPUT_FORMAT String:
   The format string defines how the programs output looks, as well as what hashes are calculated; the string must be quoted if it contains spaces.
   For each file processed, the following replacements are made in the format string, and that string is output:

• %FILEPATH% - the full path to the file including drive letter.
• %FILENAME% - the file name.
• %FILESIZE% - the file size.
• %RELATIVEPATH% - the full path to the file with the CWD removed.
• %CTIME% OR %MTIME% - the file's creation or modification time.
• %RTIME% - the run time of the program.
  "\n", "\t" or "\r"- replaced with a newline, tab, or carriage return.
  The following hash replacements will be made using lowercase hex:
  Checksums: %FLETCHER32%, %ADLER32%, %CRC32%
  MD Family: %MD2%, %MD2CNG%, %MD4%, %MD4CNG%, %MD5%, %MD5CNG%
%RIPEMD160%
• SHA-1 verions: %SHA1%, %SHA1CNG%
• SHA-2 Family: %SHA256%, %SHA256CNG%, %SHA384%, %SHA384CNG%, %SHA512%, %SHA512CNG%
• SHA-3 Family: %SHA3%, %SHA3_224%, %SHA3_256%, %SHA3_384%, %SHA3_512%
• Blake family: %BLAKE2B%, %BLAKE2B160%, %BLAKE2B256%
• Tiger family: %TIGER128%, %TIGER160%, %TIGER192% (using 0x01 padding)
  %TIGER2_128%, %TIGER2_160%, %TIGER2_192% (using 0x80 padding)
  %TIGER4_128%, %TIGER4_160%, %TIGER4_192% (tiger1 with 32 rounds)
• %WHIRLPOOL%
• Shorthand replacements: %.md5%, %.sfv%, %.sha1%, %.sha256%, or %.blake2b% will be replaced with formats matching their respective file types, including file names and newlines.

   Only the hashes specified in the format string will be calculated.
   The default format is: "%FILENAME% %MD5%\n"

Arguments are processed left to right. The subdirectory flag and the output format string can be added as many times as desired to change the processing of files/directories to their right.

If you use multisum from within a .bat file, remember you have to escape any percent signs in the args, eg: multisum.exe -o:%MD5% c:\test.txt should be multisum.exe -o:%%MD5%% c:\test.txt in a .bat file.

Other Args/Modes:

• -h prints this help text
• -b:0-3 sets the benchmark level:
  
  • -b:0 No extra output. (Default)
  • -b:1 Total run stats; reports the # of files hashed, the time & speed.
  • -b:2 File stats; also reports the time & speed for each file individually.
  • -b:3 Full stats; also runs a raw benchmark of every hash algo.
• -i:1 Engages file-based STDIN processing. On receiving this arg, the program enters into a loop processing standard input, it expects one file or folder name per line, unquoted. Files & folders will be processed as per options previously set (no options are processed while in the STDIN loop). This mode is exited when the program sees -i:0 on its own line in STDIN.
• -i:2 Processes all STDIN as a single binary data-stream to hash; hashes are printed according to the output format (minus the file-specific stuff) when there's no more stdin to read.
• -i:3 Processes STDIN in line-by-line text mode. The hashes for each line will be calcuated and output, newlines trigger the hashing but are not couted as part of the input data.
• -v:VERFICATION_FILE
  When a valid verification/checksum file is provided after the colon (again, names with spaces must be quoted), the utility will re-hash each file sepcified in the verification file and report matches and errors. Supported types are: .sfv, md5, .sha1, sha256, .blake2/.blake2b
  In verification mode, the output format flag is used to control the verification output. The following flags can be passed (eg: -o:noSummary,errorsOnly):
  
  • noSummary: will not show the finally tally sums of verified/broken files
  • noProgress: will not output the 'checking fill' lines while verification is taking place. If this flag is set, there will be no output at all until all files have been checked.
  • errorsOnly: during the list output, only outputs a list of bad files (unreadable or mismatched hashes)
  • verifiedOnly: during the list output, only outputs a list of good files (unreadable or mismatched hashes)
  • printAllLists: prints (separate) lists of both good & bad files.
  • verbose: when outputting file lists, also output the hash.
    Order, capitalization, and spacing are irrelevant when passing these flags.

Examples:
   This command runs the program on a single directory and prints the filenames & md5 hashes, one per line:
      multisum.exe "-o:%MD5% %FILENAME%\n" c:\Text_files\
   This is tab separated hashes, including subdirectories:
      multisum.exe -o:%FILENAME%\t%SHA1%\t%SHA256\n -s:1 c:\windows
   This command creates a .sfv checksum file for the current directory:
      multisum.exe -o:%.sfv% -s:1 . > checksum.sfv
   This command verifies the previously created checksum file:
      multisum.exe -v:checksum.sfv

ChangeLog

• Version 1.4.1; 2019-12-14

  • Added: Some filtering/sorting options to the verification mode. Not entirely satisfied with how these are passed, so this may change.

• Version 1.4; 2019-11-20

  • Added: Two new STDIN input modes, see above for descriptions of -i:2 and -i:3
  • Changed: Memory efficient handling of very large verification files.
  • Changed: found moderately better whirlpool implementation from splittydev (performance on my machine is now ~46 MB/s vs the old code's 12MB/s)
  • Added: New hash family: tiger.
  • Added: CNG versions of the MD and SHA hash famlies. These versions invoke direct calls to microsoft's bcrypt.dll to do the math. On many platforms this will use hardware acceleration and give a much faster result.
  • Added: New checksum, Fletcher32, like adler but better.

• Version 1.3.1; 2019-10-09

  • Changed: improved ADLER32 performance; should be faster than CRC32 in all cases now.
  • Fixed: stupid bug in new shorthand formats.

• Version 1.3; 2019-10-06

  • Added: new hashes: ADLER32, MD2, MD4, RIPEMD160, WHIRLPOOL, BLAKE2B160, BLAKE2B256. Be aware MD2 and WHIRLPOOL are glacially slow; I am looking for optimized implementions of them, or I may just remove them in the next version.
  • Added: verification mode, see info on the -v: flag above.
  • Added: more shorthand formats: %.sha1%, %.sha256%, %.blake2b%
  • Fixed: .net built-in hashes would fail to initilize on some rare platform configurations.

• Version 1.2; 2019-08-28

  • Added: %RELATIVEPATH% as an avalible replacement string.
  • Fixed: some fixes in the %.md5% and %.sfv% replacements.

• Version 1.1; 2019-08-21

  • Added: support for accepting files and folders on STDIN. See the -i flag in the usage notes above.
  • Added: support for BLAKE2B (only the 512 bit output version). Their c# codebase left a bit to be desired (started with 8 classes, ended up with 2), but I got there in the end. I'm using their fully-unrolled code for the core math; it bumps our executable size up by about 10K, but damn if it dosn't leave the SHAs in the dust.
  • Added: better support for long paths, support for hashing NTFS alternate data streams (specify an ADS like so: "file.txt:someOtherStream"). Relative paths are still limited to 260 chars. If in doubt, Prepend \\?\ to long absolute paths.
  • Added: %nullAlgo%, a dummy hash that just returns 0 in all cases. For testing IO speed.
  • Changed: now targeting .net framework version to 4.6.2 to help with long path support.

• Version 1.0; 2019-02-03

  • Initial Release.

License Information

This software includes code or resources from the following sources:

• Blake2B hashing code from the Blake2 project
  Licenced under the terms of: The CC0 public domain dedication.

• SHA3 Code by jdluzen by jdluzen
  Licensed under the terms of the The MIT License.

• Fast CRC32 Code by Max Vysokikh / force-net
  Licensed under the terms of the The MIT License.

• Quick-ish ADLER32 taken from SharpZipLib
  Licensed under the terms of the MIT License

• MD2 & MD4 implementations adapted from The Legion of the Bouncy Castle
  Licensed under the terms of: The MIT X Consortium license

• WHIRLPOOL implementation by SplittyDev
  Licensed under the terms of: "Public domain"

• CNG p/invokes collected by Andrew Arnott
  Licensed under the terms of the The MIT License.

This software is distributed as-is, without any representations or warranties of any kind.
The author of this software imposes no additional license terms or limits upon its use or redistribution.

Feedback/Bugs

Send to utils@cresstoehne.com
App Website

DocumentId: f3b1a22c3ed43443bf16cfde5adf4a54f6cd1645 EOF