Bash Remove Blank Lines from File

We can use any of the below commands if we are required to write the output to a new file. But, we can only use sed and perl commands if instructed to remove blank lines and lines containing whitespace characters and store the updated content in the original file.

Using grep Command

Use the grep command to remove blank lines and lines with whitespace characters from the specified text file in bash.

Alternatively, we can use the grep command as follows to remove blank lines and lines with whitespace characters from the specified text file in bash.

The grep command searched for the blank lines and lines with whitespace characters in the file.txt file, removed them, and wrote the updated content in the outFile.txt file. Then, the echo commands were used with the cat command to display the content of the specified file. Let’s break down the commands to understand their components below:

• The grep command is Global Regular Expression Print. A command-line utility searched the given patterns in the specified text files.
• The -v option informed the grep command to invert the match, which means printing all lines that don’t match the given pattern.
• The '^[[:space:]]*$' regex pattern matched blank lines and lines with whitespace characters. The ^ matched the start of the line, $ matched the line’s end, and [[:space:]] matched any whitespace characters (tab or space). So, the '^[[:space:]]*$' pattern matched the lines having zero or more whitespace characters between the start and end of a line. Remember to enclose the regex within the quotes.
• The '^[[:blank:]]*$' did the same as '^[[:space:]]*$'. It is just an alternate way to write regex.
• The file.txt is the input file the grep command uses to search for empty lines.
• The > is the redirection operator, which redirected the output of the grep command to the outFile.txt file. If it already exists, the> operator will overwrite the outFile.txt file. You can also use >> if you don’t want to overwrite the outFile.txt but append it.
• The $(cat file.txt) and $(cat outFile.txt) are command substitutions used to run the cat command on the specified file, capturing the output as a string. This output was printed on the terminal (bash console).
• The echo commands were used to print the customized output on the bash console.
• In the echo commands, we used the -e parameter to interpret the \n character.
• The cat command was used to retrieve (read) the content of the specified file.

Using awk Command

Use the awk command to remove blank lines and lines with whitespace characters from the specified text file in bash.

We have already learned about the echo command and its components. So here we are breaking down the awk command to understand it and its components.

• The awk is a text processing utility capable of filtering, searching and manipulating the specified text files.
• The 'NF' is an awk pattern matched any line with at least one field, which was any non-empty line. Why did it work in this way? Because, in awk, a line is split into fields based on the delimiter, it is whitespace by default. Note that the NF is a built-in variable representing the fields’ number on a given line.
• file.txt was the input file name, while outFile.txt was the output file name.

Using sed Command

Use the sed command to remove blank lines and lines with whitespace characters from the specified text file in bash.

Use the sed command to remove blank lines and lines with whitespace characters from the specified text file in bash and write the output to the original file.

We have already learned about echo and its components in the first section while learning the grep command. Let’s break down the sed command to understand it.

• The sed is a stream editor command; we used it to manipulate the specified text files.
• The /^[[:space:]]*$/ pattern matched all blank lines whether they have whitespace characters.
• The /d is a sed command, instructing it to delete the matched lines.
• The file.txt was the input file name while outFile.txt was the output file name.
• The -i option was used to update the original file, which was file.txt in the above examples. While using the -i option, the input and output file names were the same, file.txt.

Further reading:

Remove Newline from String in Bash

Read more →

Bash Remove Character from String

Read more →

Using perl Command

Use the perl command to remove blank lines and lines with whitespace characters from the specified text file in bash.

Use the perl command to remove blank lines and lines with whitespace characters from the specified text file in bash and write the output to the original file.

In the first section, we learned about echo while using the grep command. Here, we will focus on the perl command and its components; let’s break it down to understand.

• The perl is the Perl interpreter’s name; we used it to execute the Perl code.
• The -ne option was used to instruct the Perl to read the file content (the content of file.txt) line by line and execute the 'print if /\S/'code for every line.
• The 'print if /\S/' was the Perl code we ran it for every line of the file.txt file. This code snippet used the print statement to output every line of file.txt but only if it matched the given regex, \S.The \S matched one or multiple blank lines and lines containing whitespace characters.
• The file.txt was the input file name while outFile.txt was the output file name.
• The -i option told Perl to update the original file in place, which was file.txt in the above examples. While using the -i option, the input and output file names were the same, file.txt.

If your input file contains blank lines only (does not have lines containing whitespace characters), then you can use grep, tr, awk, sed and perl commands as follows; do not forget to replace the file.txt with your input file name and outFile.txt with your output file name.

• grep -v '^$' file.txt > outFile.txt
• tr -s '\n' outFile.txt
• awk 'NF' file.txt > outFile.txt
• sed '/^$/d' file.txt > outFile.txt or sed -i '/^$/d' file.txt
• perl -ne 'print unless /^$/' file.txt > outFile.txt or perl -i -ne 'print unless /^$/' file.txt

That’s all about bash remove blank lines from File.