Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
1,074
questions
271
votes
5
answers
70k
views
Why is using a shell loop to process text considered bad practice?
Is using a while loop to process text generally considered bad practice in POSIX shells?
As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
- 152k
696
votes
4
answers
310k
views
Why is printf better than echo?
I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
- 12.3k
955
votes
10
answers
1.8m
views
How can I replace a string in a file(s)?
Replacing strings in files based on certain search criteria is a very common task. How can I
replace string foo with bar in all files in the current directory?
do the same recursively for sub ...
- 238k
96
votes
15
answers
78k
views
Removing control chars (including console codes / colours) from script output
I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
- 1,071
230
votes
9
answers
303k
views
Looping through files with spaces in the names? [duplicate]
I wrote the following script to diff the outputs of two directores with all the same files in them as such:
#!/bin/bash
for file in `find . -name "*.csv"`
do
echo "file = $file";
diff $...
- 7,143
104
votes
4
answers
138k
views
Why is `while IFS= read` used so often, instead of `IFS=; while read..`?
It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
- 32.7k
94
votes
13
answers
28k
views
Linux tools to treat files as sets and perform set operations on them
Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
- 1,043
5
votes
2
answers
707
views
How to perform replacements defined in one file on another file
Preface: Every couple of days there comes a question of this type, which is easy to solve with sed, but takes time to explain. I'm writing this question and answer, so I can later refer to this ...
- 13.4k
74
votes
4
answers
37k
views
How does awk '!a[$0]++' work?
This one-liner removes duplicate lines from text input without pre-sorting.
For example:
$ cat >f
q
w
e
w
r
$ awk '!a[$0]++' <f
q
w
e
r
$
The original code I have found on the internets read:...
- 1,808
78
votes
23
answers
55k
views
Is there a robust command line tool for processing csv files?
I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
- 45.7k
91
votes
7
answers
73k
views
How can I make iconv replace the input file with the converted output?
I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT.
Since adding the -o parameter ( in my experience ) actually writes ...
- 1,055
30
votes
5
answers
6k
views
Why does shell Command Substitution gobble up a trailing newline char?
As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens
x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p
# ...
- 32.7k
220
votes
10
answers
298k
views
How to remove duplicate lines inside a text file?
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).
What I need is to remove all the repetitions ...
- 17.5k
15
votes
3
answers
28k
views
Print lines between (and excluding) two patterns
I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed
If param1 is line matching pattern from other file using sed, below command will works fine:...
- 333
166
votes
20
answers
84k
views
A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1
Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ?
I have my ...
- 32.7k
64
votes
11
answers
31k
views
Is there a way to modify a file in-place?
I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
- 973
279
votes
18
answers
301k
views
How do you sort du output by size?
How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong.
[~]# du -sh /var/* | sort -n
0 /var/mail
1.2M /...
- 58.5k
278
votes
19
answers
450k
views
How to add a newline to the end of a file?
Using version control systems I get annoyed at the noise when the diff says No newline at end of file.
So I was wondering: How to add a newline at the end of a file to get rid of those messages?
- 15.2k
46
votes
2
answers
90k
views
Remove lines based on duplicates within one column without sort
I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
- 469
32
votes
5
answers
54k
views
How to grep for text in a file and display the paragraph that has the text?
Below is the text in the file:
Pseudo name=Apple
Code=42B
state=fault
Pseudo name=Prance
Code=43B
state=good
I need to grep for "42B" and get the output from the above text like:
Pseudo name=...
- 421
21
votes
11
answers
49k
views
Transposing rows and columns
I have a file with the lines like below.
title1:A1
title2:A2
title3:A3
title4:A4
title5:A5
title1:B1
title2:B2
title3:B3
title4:B4
title5:B5
title1:C1
title2:C2
title3:C3
title4:C4
title5:C5
...
- 211
42
votes
2
answers
15k
views
How to make tr aware of non-ascii(unicode) characters?
I'm trying to remove some characters from file(UTF-8). I'm using tr for this purpose:
tr -cs '[[:alpha:][:space:]]' ' ' <testdata.dat
File contains some foreign characters (like "Латвийская" or "...
- 6,906
27
votes
4
answers
13k
views
Print lines between (and including) two patterns
I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work.
Input:
kkkkkkkkkkk
...
- 901
160
votes
16
answers
440k
views
How to insert text before the first line of a file?
I've been looking around sed command to add text into a file in a specific line.
This works adding text after line 1:
sed '1 a\
But I want to add it before line 1. It would be:
sed '0 a\
but I get ...
- 1,737
99
votes
11
answers
194k
views
How to test whether a file uses CRLF or LF without modifying it?
I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and ...
- 5,531
38
votes
9
answers
70k
views
How to remove blank lines from a file (including tab and spaces)?
I want to remove all empty lines from a file. Even if the line contains spaces or tabs it should also be removed.
25
votes
9
answers
37k
views
Turn list into single line with delimiter
I have to take a list (loads) of IP addresses in this format:
134.27.128.0
111.245.48.0
109.21.244.0
and turn them into this format with a pipe in-between (IPs made up)
134.27.128.0 | 111.245.48....
- 251
15
votes
5
answers
6k
views
What's the best way to take a segment out of a text file?
What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
120
votes
13
answers
160k
views
Replace environment variables in a file with their actual values?
Is there an easy way to substitute/evaluate environment variables in a file?
Like let's say I have a file config.xml that contains:
<property>
<name>instanceId</name>
<...
- 1,307
59
votes
11
answers
110k
views
How to insert the content of a file into another file before a pattern (marker)?
File1 contents:
line1-file1 "1"
line2-file1 "2"
line3-file1 "3"
line4-file1 "4"
File2 contents:
line1-file2 "25"
line2-file2 "24"
Pointer-file2 "23"
line4-...
- 851
33
votes
5
answers
82k
views
Multiline pattern match using sed, awk or grep [duplicate]
Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and }
So it should be able to match
1. {}
2. {.....}
3. {........
user6106
569
votes
8
answers
664k
views
Can grep output only specified groupings that match?
Say I have a file:
# file: 'test.txt'
foobar bash 1
bash
foobar happy
foobar
I only want to know what words appear after "foobar", so I can use this regex:
"foobar \(\w\+\)"
The parenthesis ...
- 18.6k
33
votes
12
answers
78k
views
Only remove commas embedded within quotes in a comma delimited file
I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row
123,"ABC, DEV 23",345,534.202,NAME
I need to ...
- 27.2k
21
votes
4
answers
113k
views
compare two columns of different files and print if it matches
I am using Solaris 10 and so grep options involving -f don't work.
I have two pipe-separated files:
file1:
abc|123|BNY|apple|
cab|234|cyx|orange|
def|kumar|pki|bird|
file 2:
abc|123|
kumar|pki|
...
- 231
41
votes
8
answers
42k
views
How to grep-inverse-match and exclude "before" and "after" lines
Consider a text file with the following entries:
aaa
bbb
ccc
ddd
eee
fff
ggg
hhh
iii
Given a pattern (e.g. fff), I would like to grep the file above to get in the output:
all_lines except (...
- 40.6k
19
votes
7
answers
5k
views
How can I prepend a tag to the beginning of several files?
I need to add PHP tags surrounding a file. It's easy to append them using
find . -exec echo "?>" >> '{}' \;
but how can I prepend the tag <?php?
user2362
10
votes
3
answers
5k
views
How to print a line if that line or the next line do not contain a particular string
Input.txt:
8B0C
remove
8B0D
remove
8B0E
remove
8B0F
8B10
remove
8B14
remove
8B15
remove
8B16
remove
8B17
remove
8AC0
8AC1
...
- 1,061
98
votes
11
answers
353k
views
Remove last character from line
I want to remove last character from a line:
[root@ozzesh ~]#df -h | awk '{ print $5 }'
Use%
22%
1%
1%
59%
51%
63%
5%
Expected result:
Use
22
1
1
59
51
63
5
- 3,609
45
votes
7
answers
47k
views
Non-greedy match with SED regex (emulate perl's .*?)
I want to use sed to replace anything in a string
between the first AB and the first occurrence of AC (inclusive)
with XXX.
For example, I have this string (this string is for a test only):
...
- 3,229
43
votes
9
answers
125k
views
How to print only last column?
echo -e 'one two three\nfour five six\nseven eight nine'
one two three
four five six
seven eight nine
how can I do some "MAGIC" do get this output?:
three
six
nine
UPDATE:
I don't need it in this ...
- 39.7k
15
votes
4
answers
24k
views
sed - Print lines matched by a pattern range if one line matches a condition [duplicate]
I have a multi line log entry format that I need to process.
The log looks something like this:
--START--
Device=B
Data=asdfasdf
Lorem=Ipsum
--END--
--START--
Device=A
Data=asdfasdf
Lorem=Ipsum
--...
- 153
12
votes
6
answers
3k
views
Search for a string and print everything before and after within a range
I have this file:
sometext1{
string1
}
sometext2{
string2
string3
}
sometext3{
string4
string5
string6
}
I want to search this file for a specific string and print everything before this string up ...
- 123
11
votes
9
answers
4k
views
How do I print all lines of a file with duplicate values in a certain column
Given a file like this:
1,768,12,46576457,7898
1,123,435,134,146
2,345,6756856,12312,1311
5,234,567465,12341,1341
1,3245,4356345,2442,13
9,423,2342,121,463
9,989,342,121,1212
I would like to list ...
- 211
206
votes
5
answers
114k
views
How can I wrap text at a certain column size?
I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
- 44.9k
202
votes
6
answers
696k
views
Return only the portion of a line after a matching pattern
So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
- 2,614
186
votes
7
answers
426k
views
How do I delete the first n lines of an ascii file using shell commands?
I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so ...
- 9,263
47
votes
2
answers
108k
views
Common lines between two files [duplicate]
I have the following code that I run on my Terminal.
LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed
This doesn't give me the common lines between the two files. ...
- 663
25
votes
3
answers
30k
views
Substitute pattern within a file with the content of other file
I have a text file (devel.xml).
I added the word REPLACETHIS to it in order to replace this string with the content within a different file (temp.txt).
The closest thing I have is this:
sed -i -e "/...
- 371
18
votes
10
answers
13k
views
Remove extra header lines from file, except for the first line
I have a file that looks like this toy example. My actual file has 4 million lines, about 10 of which I need to delete.
ID Data1 Data2
1 100 100
2 100 200
3 200 100
ID Data1 ...
- 414
10
votes
4
answers
3k
views
How to print lines number 15 and 25 out of each 50 lines?
I have a large file and I would like print from each sequential 50 lines , the 15th and 25th lines.
sed -n '15,25p' inputfile
How to modify this command to print only lines 15 and 25 and to loop ...
- 897