BASH Programming

Bash shuf command

Shuf is one of those commands that most bash programmers have not heard. For those who have, the experience itself is often eerie, like a whisper coming from a dark distant terminal where long-forgotten commands end up.  Those that venture further find shuf and are never the same again.

What is shuf?

shuf is a command-line utility like sort included in Coreutils. You may have guessed that it is used to pseudo randomize a given input in the same way you would shuffle a deck of cards. You guessed right!

Here we will cover the shuf command along with alternatives just in case you find yourself stuck without a shuf.

Where to find help on shuf

Once you know what shuf is, the next step is knowing how to use it. Like most command-line utilities, shuf comes with an –help long option.

Command

# shuf --help

Usage: shuf [OPTION]... [FILE]
or:  shuf -e [OPTION]... [ARG]...
or:  shuf -i LO-HI [OPTION]...

Write a random permutation of the input lines to standard output.
With no FILE, or when FILE is -, read standard input.
Mandatory arguments to long options are mandatory for short options too.

-e, --echo                treat each ARG as an input line
-i, --input-range=LO-HI   treat each number LO through HI as an input line
-n, --head-count=COUNT    output at most COUNT lines
-o, --output=FILE         write result to FILE instead of standard output
--random-source=FILE      get random bytes from FILE
-r, --repeat              output lines can be repeated
-z, --zero-terminated     line delimiter is NUL, not newline
--help                    display this help and exit
--version                 output version information and exit

The Three Ways of shuf

There are three ways to use the shuf command which are:

  1. file shuf
  2. list shuf
  3. range shuf

Each way has its advantages. Knowledge of all the ways beforehand may reduce the need to use other external commands in conjunction with the shuf command.

file shuf

File shuf is the most common way shuf is used in command line. When the -e or -i option are not included in the options, shuf will operate as file shuf. That is, the input to be shuffled will be a file whether standard input or any given file. The last word in the parameter list may be a filename. In case this parameter is omitted file is taken to be standard input from the shell or piped. A – may be included following the convention that standard input is inferred.

Here follow usage and ways to specify the file in file shuf.

Usage

Usage: shuf [OPTION]... [FILE]

Ways to specify a file

There is more than one way to specify the file using file shuf. Here are example commands for each way.

Implicit file as standard input

In this way we omit file from the arguments of the shuf command. Following the convention, you may opt to include a – in place of file to indicate the file to be taken as standard input.

Commands

{
    seq 3 | shuf
}

Output

1
3
2

Explicit file as standard input

Commands

{
    seq 3 | shuf -
}

Output

3
1
2

Notes

(1) Adding a – at the end of shuf serves as a visual cue improving readability in bash scripts.

File as the name of file

In this way, we specify a filename as file in the arguments of the shuf command. Here follow a few file shuf examples using files.

Shuffle input lines from the terminal

Commands

{
    shuf /dev/fd/1
}
asdf
sdf
df
f
Ctrl-D

Output

df
f
asdf
sdf

Notes

(1) The above shuf command shuf /dev/fd/1 is equivalent to shuf –

(2) Termination of input lines through Ctrl-D is required

Shuffle lines in file

Commands

{
    seq 3 > file;
    shuf file;
    rm -f file
}

Output

2
1
3

list shuf

In the last way to shuf, we operated on a file or input piped into the shuf command. In this way to shuf, we allow input lines to be specified as arguments of the shuf command using the -e option, forcing shuf to operate as list shuf.

Usage

Usage: shuf -e [OPTION]... [ARG]...

Ways to specify list args

Type input as args

Commands

{
    shuf -e 1 2 3
}

Output

1
3
2

Notes
(1) The above shuf command  shuf -e 1 2 3 is equivalent to seq 3 | shuf –

Variable as args

Commands

{
    var="1 2 3";
    shuf -e ${var}
}

<strong>Output</strong>
[cc lang="bash"]
3
1
2

Parameter expansion as args

Commands

{
    shuf -e {1..3}
}

Output

1
2
3

Command substitution as args

Commands

{
    shuf -e $( seq 3 )
}

Output

3
2
1

range shuf

This last way to shuf is unlike the previous ways introduced. Instead of specifying a file or args in the command line, it requires a range of integers. The -i option, forces shuf to operate as range shuf.
Range shuf produces a range of integers in random order.

Usage

Usage: shuf -i LO-HI [OPTION]...

Ways to specify range

The one way: LO-HI

Commands

{
    shuf -i 1-3
}

Output

2
3
1

Notes

(1) The shuf command shuf -i 1-3 is equivalent to all previous command using the sequence 1 2 3

Advanced shuf options

Here are some of the advanced options for shuf that may prove useful in bash programming.

Limit number of output lines

To limit the number of lines in the output, we use the -n option followed by an integer as follows.

Commands

{
    shuf -i 1-3 -n 1
}

Output

3

Notes

  • The shuf command shuf -i 1-3 -n 1 is equivalent to shuf -i 1-3 | head -1

Specify a file to write output lines

To specify a file to write output lines, we use the -o option followed by a filename as follows.

Commands

{
    shuf -i 1-3 -n 1 -o file;
    cat file;
    rm -f file
}

Output

1

Notes

(1) The shuf command shuf -i 1-3 -n 1 -o file is equivalent to shuf -i 1-3 -n 1 > file using I/O redirection

Stream output lines

To create a continuous stream of output lines, we use the -r option as follows.

Commands

{
    shuf -e {0,1} -r | xargs -i echo -n "{}"
}

Output

000101101010101101010110000101111010001010111001110…

Use the zero byte instead of newline as line delimiter

To use zero-terminated lines, we use the -z option as follows.

Commands

{
    seq 3 | tr '\n' '\0' | shuf -z
}

Output

213

Notes

(1) The output contains non-printing zero byte between digits

How to shuf in bash the easy way

The easy way to shuf is to use the shuf command as discussed above. However, if you are a little curious about how you would shuf otherwise using other external commands or even pure bash, read on.

How to shuf the hard way

I have compiled a list of hard ways to shuf. Don’t worry they are not that hard. They just don’t make use of the shuf command.

Using sort

A common alternative to using file shuf is to use the sort command. Here’s how:

Commands

{
    seq 3 | sort -r
}

Output

2
3
1

Notes

(1) The shuf seq 3 | sort -r is equivalent to shuf -i 1-3

(2) shuf -i 1-3 is faster

Using gawk

Another alternative to using file shuf is to use the gawk command. Here’s how:

Script

gawk-shuf() {
  gawk -v random=${RANDOM} '
  function randInt() {
    return int(rand()*1000)
  }
  function case_numeric_compare(i1, v1, i2, v2, l, r) {
    l = int(v1)
    r = int(v2)
    if(l<r) return -1
    else if(l==r) return 0
    else return 1
  }
BEGIN {
  count=1
  srand(random)
}
{
  rank[count]=randInt()
  line[count]=$(0)
  count++
}
END {
asorti(rank,order,"case_numeric_compare")
  for(i=0;i&lt;count;i++) {
    print line[order[i]]
  }
}
' -
}
if [ ${#} -eq 0 ]
then
  true
else
  exit 1 # wrong args
fi
gawk-shuf

Source:  gawk-shuf.sh

Commands

{
    seq 3 | bash gawk-shuf.sh
}

Output

2
3
1

Notes

(1) You may also use awk
(2) gawk-shuf.sh must exist in the working directory of commands

Pure bash

Script

pure-bash-shuf() {
local line
local -a lines
while read -r line
do
  lines[RANDOM]=${line}
done
for line in ${lines[@]}
do
  echo ${line}
done
}
if [ ${#} -eq 0 ]
then
  true
else
  exit 1 # wrong args
fi
pure-bash-shuf

Source: pure-bash-shuf.sh

Commands

{
    seq 3 | bash pure-bash-shuf.sh
}

Output

2
3
1

Notes

  • The above script does not handle the case in which RANDOM happens to occur more than once. That is left as an exercise.

dJackblck17 The Game

I have put together a card game start called Jackblck17 based on blackjack for the purpose of showing the shuf command in action.

Script

deck() {
    echo {A,2,3,4,5,6,7,8,9,10,J,Q,K}-{club,diamond,heart,spade}
}
banner() {
cat << EOF

bbbbbbbb
jjjj     b::::::b  lllllll  kkkkkkkk  1111111  77777777777777777777
j::::j   b::::::b  l:::::l  k::::::k  1::::::1 7::::::::::::::::::7
jjjj     b::::::b  l:::::l  k::::::k  1:::::::1 7::::::::::::::::::7
b:::::b  l:::::l   k::::::k 111:::::1  777777777777:::::::7
jjjjjjj  aaaaaaaaaaaaa  ccccccccccccccccb:::::bbbbbbbbb l::::l
cccccccccccccccc k:::::k    kkkkkkk   1::::1   7::::::7
j:::::j  a::::::::::::a   cc:::::::::::::::cb::::::::::::::bb
l::::l   cc:::::::::::::::c k:::::k   k:::::k    1::::1  7::::::7
j::::j  aaaaaaaaa:::::a c:::::::::::::::::cb::::::::::::::::b  l::::l
c:::::::::::::::::c k:::::k  k:::::k     1::::1  7::::::7
j::::j a::::ac:::::::cccccc:::::cb:::::bbbbb:::::::b l::::l
c:::::::cccccc:::::c k:::::k k:::::k   1::::l  7::::::7
j::::j    aaaaaaa:::::ac::::::c cccccccb:::::b    b::::::b l::::l c::::::c
ccccccc k::::::k:::::k       1::::l          7::::::7
j::::j  aa::::::::::::ac:::::c b:::::b   b:::::b l::::l c:::::c k:::::::::::k
1::::l         7::::::7
j::::j a::::aaaa::::::ac:::::c  b:::::b     b:::::b l::::l c:::::c
k:::::::::::k        1::::l        7::::::7
j::::ja::::a    a:::::ac::::::c   cccccccb:::::b   b:::::b l::::l c::::::c
ccccccc k::::::k:::::k       1::::l       7::::::7
j::::ja::::a    a:::::ac:::::::cccccc:::::cb:::::bbbbbb::::::bl::::::lc:::::::
cccccc:::::ck::::::k k:::::k   111::::::111   7::::::7
j::::ja:::::aaaa::::::a c:::::::::::::::::cb::::::::::::::::b l::::::l
c:::::::::::::::::ck::::::k  k:::::k  1::::::::::1  7::::::7
j::::j a::::::::::aa:::a cc:::::::::::::::cb:::::::::::::::b
l::::::l  cc:::::::::::::::ck::::::k   k:::::k 1::::::::::1 7::::::7
j::::j  aaaaaaaaaa  aaaa   ccccccccccccccccbbbbbbbbbbbbbbbb
llllllll  cccccccccccccccckkkkkkkk kkkkkkk11111111111177777777
j::::j
jjjj      j::::j
j::::jj   j:::::j
j::::::jjj::::::j
jj::::::::::::j
jjj::::::jjj
jjjjjj

EOF

}
score() {
case ${1} in
A) echo 0 ;;
[2-9]|10) echo ${1} ;;
J|Q|K) echo 10 ;;
esac
}
score-hand() {
local card
local points
local -i candidate_points
local -i aces
aces=0
for card in ${hand}
do
let points+=$( score ${card/-*/} )
test ! "${card/-*/}" = "A" || {
let aces+=1
}
done
test ! ${aces} -le 0 || {
echo ${points}
}
while [ ${aces} -gt 0 ]
do
for point in ${points}
do
new_point=""
for ace in 1 11
do
candidate_points=$(( point + ace ))
test ! ${candidate_points} -le 21 || {
echo "${candidate_points}"
new_points="${new_points} ${candidate_points}"
}
done
done
let aces-=1
points="${new_points}"
done | sort -nur | head -1
}
jackblck17() {
local deck
local card
local -i turn
local hand
deck=$( shuf -e $( deck ) )
banner
echo -e " Press enter key to continue"
read
turn=1
for card in ${deck}
do
test ! ${turn} -gt 2 || {
echo -e "\nhit or stay? (h) or s "
read
test ! "${REPLY}" = "s" || {
break
}
}
echo -e "\n Your hand: \n"
hand="${hand} ${card}"
echo " ${hand}"
test ! $( score-hand ) -gt 21 || {
echo -e "\nBust!\n"
exit
}
let turn++
sleep 1
done
echo -e "Dealer's hand: 17\n"
echo -e "Your hand: $( score-hand )\n"
test ! $( score-hand ) -gt 17 && {
echo -e "Dealer wins\n"
true
} || {
echo -e "You win!\n"
}
}
if [ ${#} -eq 0 ]
then
true
else
exit 1 # wrong args
fi
jackblck17

Source: jackblck17.sh

Commands

bash jackblck17.sh

Output

bbbbbbbb
jjjj   b::::::b   lllllll  kkkkkkkk 1111111   77777777777777777777
j::::j b::::::b   l:::::l  k::::::k 1::::::1  7::::::::::::::::::7
jjjj   b::::::b   l:::::l  k::::::k 1:::::::1 7::::::::::::::::::7
b:::::b   l:::::l  k::::::k 111:::::1 777777777777:::::::7
jjjjjjj aaaaaaaaaaaaa  ccccccccccccccccb:::::bbbbbbbbb     l::::l
cccccccccccccccc k:::::k    kkkkkkk   1::::1  7::::::7
j:::::j  a::::::::::::a  cc:::::::::::::::cb::::::::::::::bb l::::l
cc:::::::::::::::c k:::::k   k:::::k    1::::1 7::::::7
j::::j  aaaaaaaaa:::::a c:::::::::::::::::cb::::::::::::::::b  l::::l
c:::::::::::::::::c k:::::k  k:::::k     1::::1    7::::::7
j::::j           a::::ac:::::::cccccc:::::cb:::::bbbbb:::::::b l::::l
c:::::::cccccc:::::c k:::::k k:::::k      1::::l           7::::::7
j::::j    aaaaaaa:::::ac::::::c     cccccccb:::::b    b::::::b l::::l
c::::::c     ccccccc k::::::k:::::k       1::::l          7::::::7
j::::j  aa::::::::::::ac:::::c             b:::::b     b:::::b l::::l
c:::::c              k:::::::::::k        1::::l         7::::::7
j::::j a::::aaaa::::::ac:::::c             b:::::b     b:::::b l::::l
c:::::c              k:::::::::::k        1::::l        7::::::7
j::::ja::::a    a:::::ac::::::c     cccccccb:::::b     b:::::b l::::l
c::::::c     ccccccc k::::::k:::::k       1::::l       7::::::7
j::::ja::::a    a:::::ac:::::::cccccc:::::cb:::::bbbbbb::::::bl::::::lc
:::::::cccccc:::::ck::::::k k:::::k   111::::::111   7::::::7
j::::ja:::::aaaa::::::a c:::::::::::::::::cb::::::::::::::::b l::::::l
c:::::::::::::::::ck::::::k  k:::::k  1::::::::::1  7::::::7
j::::j a::::::::::aa:::a cc:::::::::::::::cb:::::::::::::::b  l::::::l
cc:::::::::::::::ck::::::k   k:::::k 1::::::::::1 7::::::7
j::::j  aaaaaaaaaa  aaaa   ccccccccccccccccbbbbbbbbbbbbbbbb
llllllll  cccccccccccccccckkkkkkkk  kkkkkkk11111111111177777777
j::::j
jjjj      j::::j
j::::jj   j:::::j
j::::::jjj::::::j
jj::::::::::::j
jjj::::::jjj
jjjjjj
Press enter key to continue
Your hand:
3-heart

Your hand:</span>
3-heart 4-spade

hit or stay? (h) or (s)

Your hand:
3-heart 4-spade 9-heart

hit or stay? (h) or s
s

Dealer's hand: 17
Your hand: 16

Dealer wins

The bottom line on shuf in bash

In this tutorial, we covered all you need to know about the shuf command and more. No longer will you have to result to do things the hard way, now that you have shuf.

To be honest, before writing this, I knew little that there was a command called shuf that could be used to randomize the order of any given input. After taking a deep dive into the shuf command for bash programming, Now, I can honestly say that it was worth it; shuf is more useful than I thought.

I hope that you enjoyed reading this as much as I enjoyed writing it and that it helps you in your career or homework. If it does, let me know.

Thanks,

About the author

Nicholas Shellabarger

Nicholas Shellabarger

A developer and advocate of shell scripting and vim. His works include automation tools, static site generators, and web crawlers written in bash. For work he tools with cloud computing, app development, and chatbots. He codes in bash, python, or php, but is open to offers.