-
Notifications
You must be signed in to change notification settings - Fork 8
Manual
- Command line
- Commands from arguments
- Commands from stdin
- Parallelism
- Dry run
- Debug logging
- Error handling
- Timeout
- Progress bar
- Specifying command and initial arguments on command line
- Reading multiple inputs
- Regular Expression
- Shell Commands
- Bash Function
$ rust-parallel --help
Execute commands in parallel
By Aaron Riekenberg <[email protected]>
https://github.com/aaronriekenberg/rust-parallel
https://crates.io/crates/rust-parallel
Usage: rust-parallel [OPTIONS] [COMMAND_AND_INITIAL_ARGUMENTS]...
Arguments:
[COMMAND_AND_INITIAL_ARGUMENTS]...
Optional command and initial arguments.
If this contains 1 or more ::: delimiters the cartesian product of arguments from all groups are run.
Options:
-d, --discard-output <DISCARD_OUTPUT>
Discard output for commands
Possible values:
- stdout: Redirect stdout for commands to /dev/null
- stderr: Redirect stderr for commands to /dev/null
- all: Redirect stdout and stderr for commands to /dev/null
-i, --input-file <INPUT_FILE>
Input file or - for stdin. Defaults to stdin if no inputs are specified
-j, --jobs <JOBS>
Maximum number of commands to run in parallel, defauts to num cpus
[default: 8]
-0, --null-separator
Use null separator for reading input files instead of newline
-p, --progress-bar
Display progress bar
-r, --regex <REGEX>
Apply regex pattern to inputs
-s, --shell
Use shell mode for running commands.
Each command line is passed to "<shell-path> <shell-argument>" as a single argument.
-t, --timeout-seconds <TIMEOUT_SECONDS>
Timeout seconds for running commands. Defaults to infinite timeout if not specified
--channel-capacity <CHANNEL_CAPACITY>
Input and output channel capacity, defaults to num cpus * 2
[default: 16]
--disable-path-cache
Disable command path cache
--dry-run
Dry run mode
Do not actually run commands just log.
--exit-on-error
Exit on error mode
Exit immediately when a command fails.
--no-run-if-empty
Do not run commands for empty buffered input lines
--shell-path <SHELL_PATH>
Path to shell to use for shell mode
[default: /bin/bash]
--shell-argument <SHELL_ARGUMENT>
Argument to shell for shell mode
[default: -c]
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
The ::: separator can be used to run the Cartesian Product of command line arguments. This is similar to the ::: behavior in GNU Parallel.
$ rust-parallel echo ::: A B ::: C D ::: E F G
A C F
A D F
A C G
B C E
A D G
A D E
B C F
A C E
B C G
B D E
B D G
B D F
$ rust-parallel echo hello ::: larry curly moe
hello larry
hello moe
hello curly
# run gzip -k on all *.html files in current directory
$ rust-parallel gzip -k ::: *.html
Run complete commands from stdin.
$ cat >./test <<EOL
echo hi
echo there
echo how
echo are
echo you
EOL
$ cat test | rust-parallel
there
are
hi
how
you
By default the number of parallel jobs to run simulatenously is the number of cpus detected at run time.
This can be override with the -j/--jobs option.
With -j5 all echo commands below run in parallel.
With -j1 all jobs run sequentially.
$ rust-parallel -j5 echo ::: hi there how are you
how
hi
you
are
there
$ rust-parallel -j1 echo ::: hi there how are you
hi
there
how
are
you
Use option --dry-run for dry run mode.
In this mode the commands that would be run are ouput as info level logs.
No commands are actually run - this is useful for testing before running a job.
$ rust-parallel --dry-run echo ::: hi there how are you
2024-02-11T19:50:53.222500Z INFO rust_parallel::command: cmd="/bin/echo",args=["hi"],line=command_line_args:1
2024-02-11T19:50:53.222527Z INFO rust_parallel::command: cmd="/bin/echo",args=["there"],line=command_line_args:2
2024-02-11T19:50:53.222534Z INFO rust_parallel::command: cmd="/bin/echo",args=["how"],line=command_line_args:3
2024-02-11T19:50:53.222547Z INFO rust_parallel::command: cmd="/bin/echo",args=["are"],line=command_line_args:4
2024-02-11T19:50:53.222554Z INFO rust_parallel::command: cmd="/bin/echo",args=["you"],line=command_line_args:5
Set environment variable RUST_LOG=debug to see debug output.
This logs structured information about command line arguments and commands being run.
Recommend enabling debug logging for all examples to understand what is happening in more detail.
$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args | head -1
2024-02-11T19:50:53.225874Z DEBUG try_main: rust_parallel::command_line_args: command_line_args = CommandLineArgs { discard_output: None, input_file: [], jobs: 8, null_separator: false, progress_bar: false, regex: None, shell: false, timeout_seconds: None, channel_capacity: 16, disable_path_cache: false, dry_run: false, exit_on_error: false, no_run_if_empty: false, shell_path: "/bin/bash", shell_argument: "-c", command_and_initial_arguments: ["echo", ":::", "hi", "there", "how", "are", "you"] }
$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args:1
2024-02-11T19:50:53.230653Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1}: rust_parallel::command: begin run
2024-02-11T19:50:53.230804Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=36918}: rust_parallel::command: spawned child process, awaiting completion
2024-02-11T19:50:53.231583Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=36918}: rust_parallel::command: command exit status = exit status: 0
2024-02-11T19:50:53.231596Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=36918}: rust_parallel::command: end run
If any commands run exit with non-0 status, this is considered a failure and an error is logged.
When rust-parallel terminates, if any command failed it will log number of failed commands and exit with status 1.
Here we try to use cat to show non-existing files A, B, and C, so each command exits with status 1:
$ rust-parallel cat ::: A B C
cat: A: No such file or directory
2024-02-11T19:50:53.235707Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat" args=["A"] line=command_line_args:1 exit_status=1
cat: B: No such file or directory
2024-02-11T19:50:53.235757Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat" args=["B"] line=command_line_args:2 exit_status=1
cat: C: No such file or directory
2024-02-11T19:50:53.235798Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat" args=["C"] line=command_line_args:3 exit_status=1
2024-02-11T19:50:53.235820Z ERROR rust_parallel: fatal error in main:
3 commands failed
The --exit-on-error option can be used immediately to exit with status 1 when the first command failure is detected.
This is a best-effort option as multiple parallel commands may be running when the first failure is noticed:
$ rust-parallel --exit-on-error cat ::: A B C
cat: A: No such file or directory
2024-02-11T19:50:53.239236Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat" args=["A"] line=command_line_args:1 exit_status=1
The -t option can be used to specify a command timeout in seconds:
$ rust-parallel -t 0.5 sleep ::: 0 3 5
2024-02-11T19:50:53.744184Z WARN rust_parallel::command: child process error command: cmd="/bin/sleep",args=["3"],line=command_line_args:2 error: timeout: deadline has elapsed
2024-02-11T19:50:53.744189Z WARN rust_parallel::command: child process error command: cmd="/bin/sleep",args=["5"],line=command_line_args:3 error: timeout: deadline has elapsed
The -p option can be used to enable a graphical progress bar.
This is best used for commands which are running for at least a few seconds, and which do not produce output to stdout or stderr.
In the below command -d all is used to discard all output from commands run:
$ rust-parallel -d all -p sleep ::: 1 2 3
⠤ [00:00:01] Commands Done/Total: 1/3 █████████░░░░░░░░░░░░░░░░░░ ETA 00:00:02
Here md5 -s will be prepended to each input line to form a command like md5 -s aal
$ head -100 /usr/share/dict/words | rust-parallel md5 -s | head -10
MD5 ("aardvark") = 88571e5d5e13a4a60f82cea7802f6255
MD5 ("A") = 7fc56270e7a70fa81a5935b72eacbe29
MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661
MD5 ("aa") = 4124bc0a9335c27f086f24ba207a4912
MD5 ("aam") = 35c2d90f7c06b623fe763d0a4e5b7ed9
MD5 ("aal") = ff45e881572ca2c987460932660d320c
MD5 ("aalii") = 0a1ea2a8d75d02ae052f8222e36927a5
MD5 ("Aani") = e9b22dd6213c3d29648e8ad7a8642f2f
MD5 ("Aaron") = 1c0a11cc4ddc0dbd3fa4d77232a4e22e
MD5 ("Aaronical") = 870e7a1e963151873a3262952c1d6164
By default rust-parallel reads input from stdin only. The -i option can be used 1 or more times to override this behavior. -i - means read from stdin, -i ./test means read from the file ./test:
$ cat >./test <<EOL
foo
bar
baz
EOL
$ head -5 /usr/share/dict/words | rust-parallel -i - -i ./test echo
aalii
a
aal
bar
baz
aa
A
foo
Regular expressions can be specified by the -r or --regex command line argument.
Named or numbered capture groups are expanded with data values from the current input before the command is executed.
In these examples using command line arguments {url} and {filename} are named capture groups. {0} is a numbered capture group.
$ rust-parallel -r '(?P<url>.*),(?P<filename>.*)' echo got url={url} filename={filename} ::: URL1,filename1 URL2,filename2
got url=URL2 filename=filename2
got url=URL1 filename=filename1
$ rust-parallel -r '(?P<url>.*) (?P<filename>.*)' echo got url={url} filename={filename} full input={0} ::: URL1 URL2 ::: filename1 filename2
got url=URL1 filename=filename1 full input=URL1 filename1
got url=URL1 filename=filename2 full input=URL1 filename2
got url=URL2 filename=filename2 full input=URL2 filename2
got url=URL2 filename=filename1 full input=URL2 filename1
In the next example input file arguments {0} {1} {2} {3} are numbered capture groups, and the input is a csv file:
$ cat >./test <<EOL
foo,bar,baz
foo2,bar2,baz2
foo3,bar3,baz3
EOL
$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo got arg1={1} arg2={2} arg3={3} full input={0}
got arg1=foo arg2=bar arg3=baz full input=foo,bar,baz
got arg1=foo2 arg2=bar2 arg3=baz2 full input=foo2,bar2,baz2
got arg1=foo3 arg2=bar3 arg3=baz3 full input=foo3,bar3,baz3
All occurrences of capture groups are replaced as exact strings. Surrounding characters have no effect on this.
This means capture groups can be nested with other { or } characters such as when building json:
$ cat >./test <<EOL
1,2,3
4,5,6
7,8,9
EOL
$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo '{"one":{1},"two":{2},"nested_object":{"three":{3}}}'
{"one":1,"two":2,"nested_object":{"three":3}}
{"one":7,"two":8,"nested_object":{"three":9}}
{"one":4,"two":5,"nested_object":{"three":6}}
Shell commands can be written using -s shell mode.
Multiline commands can be written using ;.
Environment variables, $ characters, nested commands and much more are possible:
$ rust-parallel -s -r '(?P<arg1>.*) (?P<arg2>.*)' 'FOO={arg1}; BAR={arg2}; echo "FOO = ${FOO}, BAR = ${BAR}, shell pid = $$, date = $(date)"' ::: A B ::: CAT DOG
FOO = B, BAR = DOG, shell pid = 37079, date = Sun Feb 11 13:50:53 CST 2024
FOO = A, BAR = DOG, shell pid = 37078, date = Sun Feb 11 13:50:53 CST 2024
FOO = B, BAR = CAT, shell pid = 37077, date = Sun Feb 11 13:50:53 CST 2024
FOO = A, BAR = CAT, shell pid = 37076, date = Sun Feb 11 13:50:53 CST 2024
-s shell mode can be used to invoke an arbitrary bash function.
Similar to normal commands bash functions can be called using stdin, input files, or from command line arguments.
Define a bash fuction logargs that logs all arguments and make visible with export -f:
$ logargs() {
echo "logargs got $@"
}
$ export -f logargs
$ rust-parallel -s logargs ::: A B C ::: D E F
logargs got A D
logargs got A F
logargs got C E
logargs got B D
logargs got B E
logargs got B F
logargs got A E
logargs got C D
logargs got C F
$ cat >./test <<EOL
logargs hello alice
logargs hello bob
logargs hello charlie
EOL
$ cat test | rust-parallel -s
logargs got hello charlie
logargs got hello alice
logargs got hello bob
$ cat >./test <<EOL
alice
bob
charlie
EOL
$ cat test | rust-parallel -s logargs hello
logargs got hello alice
logargs got hello bob
logargs got hello charlie