Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

Filter by
Sorted by
Tagged with
189
votes
4answers
36k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
540
votes
4answers
221k views

Why is printf better than echo?

I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
731
votes
7answers
1.5m views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
79
votes
4answers
84k views

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
140
votes
10answers
174k views

Looping through files with spaces in the names? [duplicate]

I wrote the following script to diff the outputs of two directores with all the same files in them as such: #!/bin/bash for file in `find . -name "*.csv"` do echo "file = $file"; diff $...
68
votes
12answers
41k views

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
69
votes
8answers
49k views

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
79
votes
11answers
21k views

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
46
votes
19answers
24k views

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
24
votes
5answers
5k views

Why does shell Command Substitution gobble up a trailing newline char?

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p # ...
53
votes
11answers
18k views

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
13
votes
3answers
18k views

Print lines between (and excluding) two patterns

I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed If param1 is line matching pattern from other file using sed, below command will works fine:...
194
votes
18answers
181k views

How do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M /...
15
votes
2answers
6k views

Print lines between (and including) two patterns

I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work. Input: kkkkkkkkkkk ...
123
votes
9answers
171k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
88
votes
15answers
41k views

A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1

Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ? I have my ...
17
votes
9answers
35k views

Transposing rows and columns

I have a file with the lines like below. title1:A1 title2:A2 title3:A3 title4:A4 title5:A5 title1:B1 title2:B2 title3:B3 title4:B4 title5:B5 title1:C1 title2:C2 title3:C3 title4:C4 title5:C5 ...
11
votes
5answers
3k views

What's the best way to take a segment out of a text file?

What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
26
votes
5answers
52k views

Multiline pattern match using sed, awk or grep [duplicate]

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. {........
23
votes
5answers
35k views

How to grep for text in a file and display the paragraph that has the text?

Below is the text in the file: Pseudo name=Apple Code=42B state=fault Pseudo name=Prance Code=43B state=good I need to grep for "42B" and get the output from the above text like: Pseudo name=...
16
votes
4answers
83k views

compare two columns of different files and print if it matches

I am using Solaris 10 and so grep options involving -f don't work. I have two pipe-separated files: file1: abc|123|BNY|apple| cab|234|cyx|orange| def|kumar|pki|bird| file 2: abc|123| kumar|pki| ...
15
votes
9answers
24k views

Turn list into single line with delimiter

I have to take a list (loads) of IP addresses in this format: 134.27.128.0 111.245.48.0 109.21.244.0 and turn them into this format with a pipe in-between (IPs made up) 134.27.128.0 | 111.245.48....
2
votes
1answer
124 views

How to perform replacements defined in one file on another file

Preface: Every couple of days there comes a question of this type, which is easy to solve with sed, but takes time to explain. I'm writing this question and answer, so I can later refer to this ...
29
votes
2answers
55k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
33
votes
1answer
7k views

How to make tr aware of non-ascii(unicode) characters?

I'm trying to remove some characters from file(UTF-8). I'm using tr for this purpose: tr -cs '[[:alpha:][:space:]]' ' ' <testdata.dat File contains some foreign characters (like "Латвийская" or "...
8
votes
3answers
2k views

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...
185
votes
21answers
257k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid of those messages?
64
votes
9answers
176k views

How to insert text before the first line of a file?

I've been looking around sed command to add text into a file in a specific line. This works adding text after line 1: sed '1 a\ But I want to add it before line 1. It would be: sed '0 a\ but I get ...
22
votes
7answers
51k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
24
votes
8answers
22k views

How to grep-inverse-match and exclude “before” and “after” lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except (...
9
votes
5answers
1k views

Search for a string and print everything before and after within a range

I have this file: sometext1{ string1 } sometext2{ string2 string3 } sometext3{ string4 string5 string6 } I want to search this file for a specific string and print everything before this string up ...
7
votes
7answers
1k views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
269
votes
8answers
275k views

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
9
votes
4answers
2k views

How to print lines number 15 and 25 out of each 50 lines?

I have a large file and I would like print from each sequential 50 lines , the 15th and 25th lines. sed -n '15,25p' inputfile How to modify this command to print only lines 15 and 25 and to loop ...
138
votes
6answers
72k views

How can I wrap text at a certain column size?

I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
49
votes
9answers
175k views

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5
25
votes
1answer
6k views

How to implement a horizontal cat? [duplicate]

Standard cat concatenates files line by line (row by row, if you will). I find myself needing a horizontal cat command more and more often recently; i.e. a command that takes a list of files and ...
45
votes
13answers
82k views

How to print all lines after a match up to the end of the file?

Input file1 is: dog 123 4335 cat 13123 23424 deer 2131 213132 bear 2313 21313 I give the match the pattern from in other file ( like dog 123 4335 from file2). I match the pattern of the line is ...
30
votes
10answers
55k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...
12
votes
2answers
5k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
18
votes
7answers
3k views

How can I prepend a tag to the beginning of several files?

I need to add PHP tags surrounding a file. It's easy to append them using find . -exec echo "?>" >> '{}' \; but how can I prepend the tag <?php?
17
votes
9answers
6k views

Remove extra header lines from file, except for the first line

I have a file that looks like this toy example. My actual file has 4 million lines, about 10 of which I need to delete. ID Data1 Data2 1 100 100 2 100 200 3 200 100 ID Data1 ...
17
votes
7answers
1k views

How to process a multi column text file to get another multi column text file?

I have a text file: a aa aaa b bb bbb c cc ccc d dd ddd e ee eee f ff fff g gg ggg h hh hhh i ii iii j jj jjj How can I process it and get a 2 ...
7
votes
3answers
2k views

String replacement using a dictionary

What is a good way to do string replacements in a file using a dictionary with a lot of substituend-substituent pairs? And by a lot, I actually mean about 20 – not much, but many enough so that I want ...
5
votes
1answer
18k views

Appending word at the end of line with sed [duplicate]

I have a code block as follows in a file: BalancerMember http://snode1:80 min=1 max=1000 loadfactor=1 timeout=30 keepalive=on route=tc_snode1 BalancerMember http://snode2:80 min=1 max=1000 ...
284
votes
10answers
699k views

How to append multiple lines to a file

I am writing a bash script to look for a file if it doesn't exist then create it and append this to it: Host localhost ForwardAgent yes So "line then new line 'tab' then text" I think its a ...
65
votes
7answers
54k views

Show all the file up to the match

grep --before-context 5 shows 5 lines before the match. I want to show everything before the match. Doing grep --before-context 99999999 would work but it is not very... professional. How to show ...
69
votes
5answers
163k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
34
votes
4answers
36k views

Replace environment variables in a file with their actual values?

Is there an easy way to substitute/evaluate environment variables in a file? Like let's say I have a file config.xml that contains: <property> <name>instanceId</name> <...
14
votes
7answers
43k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are a ...