Benchmarks

Test System

• 2017 MacBook Pro (my personal laptop)
• 2.8 GHz Quad-Core Intel Core i7
  • 4 physical cores: hw.physicalcpu: 4
  • 2 threads per core, 8 threads total: hw.logicalcpu: 8
  • rust-parallel uses 8 parallel jobs by default on this machine
• 16GB RAM
• MacOS 13.3.1

TLDR

• rust-parallel is more than an order of magnitude faster than GNU Parallel in tests below (~13-14 times faster)
• rust-parallel is ~30% faster than command line xargs with 8 processes.

1000 invocations of md5 -s

rust-parallel 0.439 seconds real time elapsed (~31% faster than xargs -P8, ~13x faster than GNU Parallel)

$ rust-parallel -V
rust-parallel 0.5.3

$ time head -1000 /usr/share/dict/words | rust-parallel md5 -s > file

rust-parallel md5 -s > file  0.87s user 2.09s system 675% cpu 0.439 total

xargs with 8 processes 0.633 seconds real time elapsed

$ time head -1000 /usr/share/dict/words | xargs -P8 -n1 md5 -s > file

xargs -P8 -n1 md5 -s > file  0.65s user 1.60s system 356% cpu 0.633 total

GNU Parallel 5.755 seconds real time elapsed

$ parallel -V
GNU parallel 20230322

$ time head -1000 /usr/share/dict/words | parallel md5 -s > file

parallel md5 -s > file  4.48s user 7.73s system 212% cpu 5.755 total

10,000 invocations of md5 -s

rust-parallel 4.249 seconds real time elapsed (~32% faster than xargs -P8, ~14x faster than GNU Parallel)

$ rust-parallel -V
rust-parallel 0.5.3

$ time head -10000 /usr/share/dict/words | rust-parallel md5 -s > file

rust-parallel md5 -s > file  8.59s user 20.64s system 687% cpu 4.249 total

xargs with 8 processes 6.210 seconds real time elapsed

$ time head -10000 /usr/share/dict/words | xargs -P8 -n1 md5 -s > file

xargs -P8 -n1 md5 -s > file  6.49s user 15.84s system 359% cpu 6.210 total

GNU Parallel 60.13 seconds real time elapsed

$ parallel -V
GNU parallel 20230322

$ time head -10000 /usr/share/dict/words | parallel md5 -s > file

parallel md5 -s > file  45.85s user 81.78s system 212% cpu 1:00.13 total