Jake Wharton

Peeking at command-line ANSI escape sequences

28 October 2020

Command-line programs use color to convey additional information and to look pretty. For example, compare the output of ls with and without the --color flag:

The output of 'ls' and 'ls --color' in a folder with three entries, the latter command using color to distinguish an executable and folder from a regular file.

The color helps convey information in this compact output that would otherwise only be available in more verbose forms (-l).

In addition to color, a program may update existing output. You can see this when updating images with docker-compose:

The output of 'docker-compose' showing three lines of progress bars updating individually.

Both of these effects are created using something called ANSI escape sequences.

ANSI escape crash course

Reading the Wikipedia entry on ANSI escapes is a great starting point for learning how to recreate these examples. Each escape sequence starts with a 0x1B (escape) character followed usually by [ and then one or more commands using letters or numbers.

The ls example above uses green and blue text as well as making the colored entries bold which we can recreate.

echo -e "\e[1;32mbinary\e[0m  file  \e[1;34mfolder\e[0m"

The output of the 'echo' command replicating the output of 'ls'.

Let’s break down the interesting parts:

The docker-compose example moves the cursor to rewrite previous output which we can begin to recreate.

echo "Pulling zulu-jdk-15 ... downloading" && \
echo "Pulling zulu-jdk-11 ... downloading" && \
echo "Pulling zulu-jdk-8  ... downloading" && \
sleep 2 && \
echo -e "\e[2A\e[24C\e[32mdone\e[0m\e[K" && \
sleep 1 && \
echo -e "\e[24C\e[32mdone\e[0m\e[K" && \
sleep 1 && \
echo -e "\e[3A\e[24C\e[32mdone\e[0m\e[K\n\n"

The output of the 'echo' commands replicating the output of 'docker-compose'.

Let’s break down the interesting parts for this example:

With these ANSI escape sequences we can recreate existing programs and being to create our own. But how do we know whether we’re using the same techniques as these programs? And if we don’t know how to produce a particular output how can we discover how it was created?

Displaying ANSI sequences

Given that ANSI sequences start with the 0x1B character and then [ we can replace that escape with something else to disable it.

ls --color | sed -r 's/\x1b\[/\\e\[/g'

The output of 'ls' piped through 'sed' replacing ANSI escapes with printable characters

The sed command1 matches 0x1B and [ and replaces it with \e[ which is shown as normal text. This particular replacement is convenient because you can copy the output into an echo and see the rendered form.

In this output we can see ls is using almost exactly the same ANSI sequences as we were. The only addition is that they start with \e[0m in order to clear any existing formatting.

You may also notice that the output has changed to list each entry on its own line rather than on a single line. This is because ls detects that its output is going into a pipe rather than to a terminal display. Programs may also choose to omit color when piped which defeats the whole purpose of adding the sed command. To solve both cases, run the program using unbuffer before piping.

unbuffer ls --color | sed -r 's/\x1b\[/\\e\[/g'

The output of 'ls' with 'unbuffer' piped through 'sed'

With the pipe usage hidden by unbuffer, the output of ls is back to being a single line.

If you run docker-compose with unbuffer and piping to sed the result is clearly not correct:

unbuffer docker-compose pull | sed -r 's/\x1b\[/\\e\[/g'

The output of 'docker-compose' with 'unbuffer' piped through 'sed'

This is because docker-compose is using carriage returns (\r) to move the cursor back to column 0 on a line. We can update our sed to include a command to escape carriage returns too.

unbuffer docker-compose pull | sed -r -e 's/\x0d/\\r/g' -e 's/\x1b\[/\\e\[/g'

The output of 'docker-compose' now with carriage return escaping

Now we can see all the commands. There is a lot of output here because docker-compose is updating the display very rapidly. Unlike our toy version above, each line is fully rewritten for each update. At the very end, though, you can see the \e[32mdone\e[0m sequence as part of updating the “zulu-jdk-15” line.

Bonus technique: Asciinema

Asciinema can also be used to inspect ANSI sequences, carriage returns, and everything else that a program outputs. Every terminal image and animation captured in this post was captured using Asciinema before being fed to svg-term.

For example, the docker-compose output can be captured like this:

asciinema rec -c "docker-compose pull" docker.json

Using 'asciinema' to capture the output of the 'docker-compose' command

(Yes, I captured the above example of using Asciinema inside Asciinema!)

The resulting docker.json contains a series of JSON objects which describe the output commands.

{"version": 2, "width": 122, "height": 48, "timestamp": 1603858671, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}}
[0.412745, "o", "Pulling zulu-jdk-15 ... \r\r\nPulling zulu-jdk-11 ... \r\r\nPulling zulu-jdk-8  ... \r\r\n"]
[0.671883, "o", "\u001b[1A\u001b[2K\rPulling zulu-jdk-8  ... pulling from azul/zulu-openjdk\r\u001b[1B"]
[0.672048, "o", "\u001b[1A\u001b[2K\rPulling zulu-jdk-8  ... digest: sha256:13d16ca0335fbe1df3...\r\u001b[1B"]
[0.672159, "o", "\u001b[1A\u001b[2K\rPulling zulu-jdk-8  ... status: image is up to date for a...\r\u001b[1B"]
[0.672478, "o", "\u001b[1A\u001b[2K\r"]
[0.672507, "o", "Pulling zulu-jdk-8  ... \u001b[32mdone\u001b[0m\r\u001b[1B"]
[0.782864, "o", "\u001b[2A\u001b[2K\rPulling zulu-jdk-11 ... pulling from azul/zulu-openjdk\r\u001b[2B"]
[0.782985, "o", "\u001b[2A\u001b[2K\r"]
[0.78307, "o", "Pulling zulu-jdk-11 ... digest: sha256:315e0a2a7b6bcc2343...\r\u001b[2B"]
[0.783146, "o", "\u001b[2A\u001b[2K\rPulling zulu-jdk-11 ... status: image is up to date for a...\r\u001b[2B"]
[0.783372, "o", "\u001b[2A\u001b[2K\r"]
[0.783428, "o", "Pulling zulu-jdk-11 ... \u001b[32mdone\u001b[0m\r\u001b[2B"]
[1.091186, "o", "\u001b[3A\u001b[2K\rPulling zulu-jdk-15 ... pulling from azul/zulu-openjdk\r\u001b[3B"]
[1.09136, "o", "\u001b[3A\u001b[2K\rPulling zulu-jdk-15 ... digest: sha256:bf2d25e46d2c9fc373...\r\u001b[3B"]
[1.091511, "o", "\u001b[3A\u001b[2K\r"]
[1.091571, "o", "Pulling zulu-jdk-15 ... status: image is up to date for a...\r\u001b[3B"]
[1.091859, "o", "\u001b[3A\u001b[2K\rPulling zulu-jdk-15 ... \u001b[32mdone\u001b[0m\r"]
[1.091919, "o", "\u001b[3B"]

For a complex output like docker-compose the JSON form can be easier to understand. One other advantage is that each individual write to standard out gets its own line whereas with the sed escape technique we don’t differentiate individual writes.

If you use tools like Docker, Gradle, Bazel, and even just ls you may be familiar with seeing colored and updating output daily. By using tools like sed and asciinema you can learn how those tools render their output. Should you find yourself building a command-line tool in the future, knowledge of how to use these ANSI sequences can help delight your users–even if it’s only yourself!

  1. If you are on Mac OS, you’ll need GNU sed for the -r flag which can be installed via brew install gnu-sed and then used as gsed or by alias sed=gsed

— Jake Wharton