Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

307
votes
5answers
117k views

Why is printf better than echo?

I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
268
votes
6answers
517k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
96
votes
5answers
75k views

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
94
votes
14answers
107k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid of those messages?
93
votes
13answers
63k views

how do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M /...
76
votes
3answers
8k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
69
votes
10answers
84k views

Looping through files with spaces in the names?

I wrote the following script to diff the outputs of two directores with all the same files in them as such: #!/bin/bash for file in `find . -name "*.csv"` do echo "file = $file"; diff $...
68
votes
6answers
58k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
64
votes
6answers
25k views

How can I wrap text at a certain column size?

I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
54
votes
4answers
38k views

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
46
votes
2answers
286k views

Using 'sed' to find and replace [duplicate]

I know this question has probably been answered before. I have seen many threads about this in various places, but the answers are usually hard to extract for me. I am looking for help with an ...
46
votes
9answers
12k views

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
42
votes
3answers
89k views

How to merge all (text) files in a directory into one?

I've got 14 files all being parts of one text. I'd like to merge them into one. How to do that?
42
votes
4answers
13k views

sort inplace - like sed --in-place - exists?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
40
votes
17answers
99k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
39
votes
14answers
15k views

A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1

Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ? I have my ...
37
votes
7answers
5k views

Is there a tool that combines zcat and cat transparently?

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this: $ zcat * you end up with a command line like zcat xyz.log xyz.log.1 ...
36
votes
5answers
23k views

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
35
votes
9answers
9k views

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
34
votes
13answers
16k views

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
34
votes
9answers
19k views

sort but keep header line at the top

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
34
votes
5answers
26k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
34
votes
8answers
18k views

How to obtain inverse behavior for `tail` and `head`?

Is there a way to head/tail a document and get the reverse output; because you don't know how many lines there are in a document? I.e. I just want to get everything but the first 2 lines of foo.txt ...
33
votes
2answers
115k views

Return only the portion of a line after a matching pattern

So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
31
votes
6answers
21k views

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
29
votes
12answers
25k views

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through unix.stackexchange.com and on the ...
28
votes
4answers
26k views

Print file content without the first and last lines

Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
28
votes
4answers
66k views

How can I find out how many lines a text file contains without viewing it?

How can I find how many lines a text file contains without opening the file in an editor or a viewer application? Is there a handy Unix console command to see the number?
28
votes
3answers
80k views

remove particular characters from a variable using bash

I want to parse a variable (in my case it's development kit version) to make it dot(.) free. If version='2.3.3', desired output is 233. I tried as below, but it requires . to be replaced with another ...
28
votes
3answers
80k views

How do I delete the first n lines of an ascii file using shell commands?

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so ...
27
votes
7answers
32k views

how do I trim each line in bash?

I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into?
27
votes
2answers
8k views

How can I use column to delimit on tabs and not spaces?

I'd like to use Unix column to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I make column ...
25
votes
4answers
20k views

How to get only the unique results without having to sort data?

$ cat data.txt aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $ The result that I need is to display ...
25
votes
3answers
44k views

How to print only the duplicate values from a text file?

Suppose there is a column of numeric values like following: File1: 1 2 3 3 3 4 4 4 5 6 I want the output: 3 4 That is, only the repeated lines. Is there any command line tools to find this ...
25
votes
12answers
12k views

How can I write to the second line of a file from the command line?

I have an external program that produces an output file (largish, 20K lines possible). I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
24
votes
5answers
86k views

How to strip multipe spaces to one using sed?

sed on AIX is not doing what I think it should. I'm trying to replace multiple spaces with a single space in the output of IOSTAT: # iostat System configuration: lcpu=4 drives=8 paths=2 vdisks=0 tty:...
24
votes
6answers
62k views

combine text files column-wise

I have two text files. The first one has content: Languages Recursively enumerable Regular while the second one has content: Minimal automaton Turing machine Finite I want to combine them into ...
24
votes
6answers
78k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": [],...
24
votes
2answers
1k views

How to merge two text files into one file without using cat or sed [duplicate]

I have two files File1: a b c File2: 1 2 3 now I need to combine them to one csv file a;1 b;2 c;3 As the files are really huge, I would rather not use cat and sed to process the second file. ...
23
votes
5answers
45k views

Shell: How to read the bytes of a binary file and print as hexadecimal?

In shell, how can I read the bytes of a binary file I have, and print the output as hexadecimal numbers?
23
votes
4answers
49k views

Grabbing the first [x] characters for a string from a pipe

If I have really long output from a command (single line) but I know I only want the first [x] (let's say 8) characters of the output, what's the easiest way to get that? There aren't any delimiters.
23
votes
3answers
30k views

How to find unmatched brackets in a text file?

Today I learned that I can use perl -c filename to find unmatched curly brackets {} in arbitrary files, not necessarily Perl scripts. The problem is, it doesn't work with other types of brackets () [] ...
22
votes
9answers
5k views

How to delete every second line from a file?

File: Data inserted into table. Total count 13 No error occurred Data inserted into table. Total count 45 No error occurred Data inserted into table. Total count 14 No error occurred Data inserted ...
21
votes
11answers
33k views

How to print all lines after a match up to the end of the file?

Input file1 is: dog 123 4335 cat 13123 23424 deer 2131 213132 bear 2313 21313 I give the match the pattern from in other file ( like dog 123 4335 from file2). I match the pattern of the line ...
21
votes
5answers
25k views

Convert file contents to lower case

I have temp file with some lower-case and upper-case contents. Input Contents of my temp file: hi Jigar GANDHI jiga I want to convert all upper to lower. Command I tried the following command: ...
21
votes
2answers
2k views

Pipes, how do data flow in a pipeline?

I don't understand how the data flows in the pipeline and hope someone could clarify what is going on there. I thought a pipeline of commands processes files (text, arrays of strings) in line by line ...
21
votes
5answers
6k views

Command line friendly spreadsheets

Does such a thing exist? Text-based spreadsheets that display well in a CLI environment. I'm aware that I could cat foobar.csvand do as I please, but it isn't particularly practical or attractive. I ...
20
votes
4answers
42k views

Delete last line from the file

I use sed to quickly delete lines with specific position as sed '1d' sed '5d' But, what if I want to delete the last line of the file and I don't know the count of lines (I know I can get that ...
20
votes
2answers
75k views

How to insert text after a certain string in a file?

Right now I'm using echo "Hello World" >> file.txt to append some text to a file but I also need to add text below a certain string let's say "[option]", is it possible with sed? EG: Input ...
20
votes
7answers
43k views

How to convert TXT to PDF?

I want to convert .txt files to .pdf. I'm using this: ls | while read ONELINE; do convert -density 400 "$ONELINE" "$(echo "$ONELINE" | sed 's/.txt/.pdf/g')"; done But this produces one "error" -- ...