SREP (formerly SuperREP)

SREP is a useful command line utility that supports dictionaries larger than RAM available and helps you to process files that are 20x larger than the RAM size.

Default settings (-l512) allows to process files that are 20x larger than RAM size. Memory requirements are proportional to 1/L, so by increasing -l value it's possible to process even larger files.

When compressing, memory usage for hash: filesize/L*20 + filesize/L*4 + roundup(filesize/L*4 * 5/4) + roundup(filesize/(L*8) * 4) where roundup() rounds up to the next power of 2 and first memory block (i.e. *20) used only in -m1 mode. Overall, hash includes 3 or 4 chunks. Besides hash, compression uses 3*8mb buffers.

Decompression uses only 2*8mb buffers and no hash. Repeated data copied directly from output file, though, so you need to have enough RAM used for disk cache in order to make decompression fast.