Manipulation of text by programs, scripts, etc.
78
votes
5answers
20k views
Why is printf better than echo?
I have heard that printf is better than echo and I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL ...
30
votes
3answers
9k views
How to remove duplicate lines inside a text file?
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).
What I need is to remove all the repetitions ...
24
votes
8answers
3k views
how do you sort du output by size?
How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong.
[~]# du -sh /var/* | sort -n
0 /var/mail
1.2M ...
21
votes
6answers
2k views
Linux tools to treat files as sets and perform set operations on them
Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
18
votes
6answers
2k views
Is there a way to modify a file in-place?
I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
17
votes
4answers
5k views
Why is `while IFS= read` used so often, instead of `IFS=; while read..`?
It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
17
votes
4answers
3k views
How to obtain inverse behavior for `tail` and `head`?
Is there a way to head/tail a document and get the reverse output; because you don't know how many lines there are in a document?
I.e. I just want to get everything but the first 2 lines of foo.txt ...
17
votes
9answers
3k views
How can I write to the second line of a file from the command line?
I have an external program that produces an output file (largish, 20K lines possible).
I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
16
votes
7answers
20k views
Looping through files with spaces in the names?
I wrote the following script to diff the outputs of two directores with all the same files in them as such:
#!/bin/bash
for file in `find . -name "*.csv"`
do
echo "file = $file";
diff ...
16
votes
1answer
502 views
How to implement a horizontal cat?
Standard cat concatenates files line by line (row by row, if you will). I find myself needing a horizontal cat command more and more often recently; i.e. a command that takes a list of files and ...
15
votes
3answers
824 views
Looking for an old classical Unix toolkit textbook
I am looking for a book about the Unix command-line toolkit (sh, grep, sed, awk, cut, etc.) that I read some time ago. It was an excellent book, but I totally forgot its name. The great thing about ...
14
votes
2answers
1k views
Horizontal file concatenation
Is there a Linux command like cat that joins files with the same number of lines horizontally?
14
votes
5answers
1k views
Command line friendly spreadsheets
Does such a thing exist? Text-based spreadsheets that display well in a CLI environment. I'm aware that I could cat foobar.csvand do as I please, but it isn't particularly practical or attractive. I ...
13
votes
8answers
14k views
How to count the number of a specific character in each line?
I was wondering how to count the number of a specific character in each line by some text processing utilities?
For example, to count " in each line of the following text
"hello!"
Thank you!
The ...
13
votes
4answers
14k views
How to add a newline to the end of a file?
Using version control systems I get annoyed at the noise when the diff says No newline at end of file.
So I was wondering: How to add a newline at the end of a file to get rid off those messages?
13
votes
5answers
3k views
sort but keep header line in the at the top?
I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
13
votes
8answers
2k views
Is there a robust command line tool for processing csv files?
I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
13
votes
2answers
4k views
How can I make iconv replace the input file with the converted output?
I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT.
Since adding the -o parameter ( in my experience ) actually writes ...
13
votes
3answers
10k views
How to merge all (text) files in a directory into one?
I've got 14 files all being parts of one text. I'd like to merge them into one. How to do that?
12
votes
9answers
1k views
There must be a better way to replace single newlines only?
I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
12
votes
2answers
809 views
How can I count the number of different characters in a file?
I would need a program, that outputs the number of the different characters in a file. Example:
> stats testfile
' ': 207
'e': 186
'n': 102
Exists any tool, that do this?
12
votes
3answers
4k views
Can grep output only specified groupings that match?
Say I have a file:
# file: 'test.txt'
foobar bash 1
bash
foobar happy
foobar
I only want to know what words appear after "foobar", so I can use this regex:
"foobar \(\w\+\)"
The parenthesis ...
12
votes
2answers
5k views
How do I remove certain lines (using line numbers) in a file?
There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
11
votes
2answers
654 views
echo a file without the first and last lines
Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
11
votes
6answers
1k views
Command to display first few and last few lines of a file
I have a file with many rows, and each row has a timestamp at the starting, like
[Thread-3] (21/09/12 06:17:38:672) logged message from code.....
So, I frequently check 2 things from this log file.
...
11
votes
3answers
370 views
How can I redirect matching lines to a file, and non-matching lines to a different file?
Is there a script/program/utility already available for the following requirement in a optimised way?
someCommand |
tee >(grep "pattern" > LinesWhichMatch) |
grep -v "pattern" > ...
11
votes
3answers
3k views
Removing control chars (including console codes / colours) from script output
I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
11
votes
6answers
10k views
How to cut part from log file?
I have 8 Gb long log file (Rails production log), and I need to cut part between some dates (lines). Which command i have to use, to do this?
11
votes
4answers
269 views
How to do a continous 'wc -l' with gnu texttools?
I know of course that
cat logfile.txt | wc -l
120
will tell me the number of lines in a file.
Whereas
tail -f logfile.txt
will show me the new lines that another program writes to logfile.txt.
...
11
votes
2answers
3k views
Filtering invalid utf8
I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
10
votes
4answers
760 views
How did they manage to drive a Unix computer before mice and copy & paste?
Since Unix is 40 years old, Unix is older than the invention of the computer mouse. (actually only 3 years if Unix is from 1969 and the mouse from 1972) How in the world did a new user do anything ...
10
votes
5answers
7k views
combine text files column-wise
I have two text files. The first one has content:
Languages
Recursively enumerable
Regular
while the second one has content:
Minimal automaton
Turing machine
Finite
I want to combine them into ...
10
votes
2answers
2k views
How to split stdout to go to several output files?
Say, I have a command command which prints huge number of lines to stdout:
line1
line2
.....
lineN
I want to save the output to disk, but not as a single file, but as a sequence of files each ...
10
votes
2answers
897 views
How to get only the unique results without having to sort data?
$ cat data.txt
aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb
$ cat data.txt | uniq
aaaaaa
cccccc
aaaaaa
bbbbbb
$ cat data.txt | sort | uniq
aaaaaa
bbbbbb
cccccc
$
The result that I need is to display ...
10
votes
2answers
484 views
Where has the trailing newline char gone from my command substitution?
The following code best describes the situation. Why is the last line not outputting the trailing newline char? Each line's output is shown in the comment. I'm using GNU bash, version 4.1.5
...
9
votes
11answers
3k views
How can I split a text file into multiple text files?
I have a text file called entry.txt that contains the following:
[ entry1 ]
1239 1240 1242 1391 1392 1394 1486 1487 1489 1600
1601 1603 1657 1658 1660 2075 2076 2078 2322 2323
2325 2740 2741 2743 ...
9
votes
4answers
1k views
Delete last line from the file
I use sed to quickly delete lines with specific position as
sed '1d'
sed '5d'
But, what if I want to delete the last line of the file and I don't know the count of lines (I know I can get that ...
9
votes
2answers
1k views
How can I find out how many lines a text file contains without viewing it?
How can I find how many lines a text file contains without opening the file in an editor or a viewer application? Is there a handy Unix console command to see the number?
9
votes
6answers
912 views
How can I prepend a tag to the beginning of several files?
I need to add PHP tags surrounding a file. It's easy to append them using
find . -exec echo "?>" >> '{}' \;
but how can I prepend the tag <?php?
9
votes
5answers
6k views
Shell: How to read the bytes of a binary file and print as hexadecimal?
In shell, how can I read the bytes of a binary file I have, and print the output as hexadecimal numbers?
9
votes
2answers
2k views
How to remove multiple lines per occurrence in a file?
Say I have this 857835 line file, containing stuff like this:
a1
rubbish1
rubbish2
rubbish3
rubbish4
a1
rubbish5
rubbish6
rubbish7
rubbish8
And I wish to remove all occurences of a1 and the next ...
9
votes
5answers
227 views
How to count rows ordered by the first field in bash
Here is a snippet from the INPUT:
...
####################
Bala Bela;XXXXXX12345;XXXXXX12345678;A
SERVER345Z3.DOMAIN.com0
SERVER346Z3.DOMAIN.com0
SERVER347Z3.DOMAIN.com0
SERVER348Z3.DOMAIN.com0
...
9
votes
3answers
404 views
Doing two things with output from a command
I have a program texcount that outputs the number of words in my LaTeX document. I can also pipe the output of this to sed to make the newlines TeX linebreaks and write this to a file which I can then ...
9
votes
6answers
600 views
Text Manipulation Across multiple lines
I Have a file that has text like this:
AAAA
BBBB
CCCC
DDDD
1234
5678
9012
3456
EEEE
7890
etc...
And i want to match up the Alphabetic lines with the Numeric lines so they are like this:
...
9
votes
4answers
259 views
Delete whitespace for a set of lines in Vim editor
I have some text like the following in a file:
sample text
some random text
even more random text
text with no indent
worst indention
I need to delete the empty space before each ...
9
votes
3answers
771 views
Parsing log files for frequent IP's
So, I hacked this together while undergoing a DDOS attack to pull naughty ips out of my logs. Anyone have any improvements or other suggestions to make it better?
Here's the general idea:
pull ...
9
votes
2answers
1k views
Count how many times each line appears in a file
Say I have a file which contains:
A
A
A
B
CC
I want to have the output like this:
A 3
B 1
CC 1
9
votes
3answers
355 views
Process last line first using awk
I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally.
The ...
9
votes
5answers
1k views
Best way to follow a log and execute a command when some text appears in the log
I have a server log that outputs a specific line of text into its log file when the server is up. I want to execute a command once the server is up, and hence do something like the following:
tail -f ...
9
votes
3answers
293 views
Deleting all C comments with sed
I am trying to write a script that will delete all comments and everything in between inside C files in my current directory. I've been using sed, and this is what I have so far:
sed -i '/ * [^()] ...
