Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

Filter by
Sorted by
Tagged with
271 votes
5 answers

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
cuonglm's user avatar
  • 152k
696 votes
4 answers

Why is printf better than echo?

I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
amphibient's user avatar
  • 12.3k
955 votes
10 answers

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
terdon's user avatar
  • 238k
96 votes
15 answers

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
andrew cooke's user avatar
  • 1,071
230 votes
9 answers

Looping through files with spaces in the names? [duplicate]

I wrote the following script to diff the outputs of two directores with all the same files in them as such: #!/bin/bash for file in `find . -name "*.csv"` do echo "file = $file"; diff $...
Amir Afghani's user avatar
  • 7,143
104 votes
4 answers

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
Peter.O's user avatar
  • 32.7k
94 votes
13 answers

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
nilton's user avatar
  • 1,043
5 votes
2 answers

How to perform replacements defined in one file on another file

Preface: Every couple of days there comes a question of this type, which is easy to solve with sed, but takes time to explain. I'm writing this question and answer, so I can later refer to this ...
Philippos's user avatar
  • 13.4k
74 votes
4 answers

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets read:...
Alexander Shcheblikin's user avatar
78 votes
23 answers

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
Steven D's user avatar
  • 45.7k
91 votes
7 answers

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
meder omuraliev's user avatar
30 votes
5 answers

Why does shell Command Substitution gobble up a trailing newline char?

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p # ...
Peter.O's user avatar
  • 32.7k
220 votes
10 answers

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
Ivan's user avatar
  • 17.5k
15 votes
3 answers

Print lines between (and excluding) two patterns

I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed If param1 is line matching pattern from other file using sed, below command will works fine:...
lokomika's user avatar
  • 333
166 votes
20 answers

A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1

Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ? I have my ...
Peter.O's user avatar
  • 32.7k
64 votes
11 answers

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
Nim's user avatar
  • 973
279 votes
18 answers

How do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M /...
xenoterracide's user avatar
278 votes
19 answers

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid of those messages?
k0pernikus's user avatar
  • 15.2k
46 votes
2 answers

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
Zach C's user avatar
  • 469
32 votes
5 answers

How to grep for text in a file and display the paragraph that has the text?

Below is the text in the file: Pseudo name=Apple Code=42B state=fault Pseudo name=Prance Code=43B state=good I need to grep for "42B" and get the output from the above text like: Pseudo name=...
Jaya William's user avatar
21 votes
11 answers

Transposing rows and columns

I have a file with the lines like below. title1:A1 title2:A2 title3:A3 title4:A4 title5:A5 title1:B1 title2:B2 title3:B3 title4:B4 title5:B5 title1:C1 title2:C2 title3:C3 title4:C4 title5:C5 ...
Dens's user avatar
  • 211
42 votes
2 answers

How to make tr aware of non-ascii(unicode) characters?

I'm trying to remove some characters from file(UTF-8). I'm using tr for this purpose: tr -cs '[[:alpha:][:space:]]' ' ' <testdata.dat File contains some foreign characters (like "Латвийская" or "...
MatthewRock's user avatar
  • 6,906
27 votes
4 answers

Print lines between (and including) two patterns

I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work. Input: kkkkkkkkkkk ...
Rana Khan's user avatar
  • 901
160 votes
16 answers

How to insert text before the first line of a file?

I've been looking around sed command to add text into a file in a specific line. This works adding text after line 1: sed '1 a\ But I want to add it before line 1. It would be: sed '0 a\ but I get ...
Manolo's user avatar
  • 1,737
99 votes
11 answers

How to test whether a file uses CRLF or LF without modifying it?

I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and ...
Adam Ryczkowski's user avatar
38 votes
9 answers

How to remove blank lines from a file (including tab and spaces)?

I want to remove all empty lines from a file. Even if the line contains spaces or tabs it should also be removed.
Jamshed Ansari user3000272's user avatar
25 votes
9 answers

Turn list into single line with delimiter

I have to take a list (loads) of IP addresses in this format: and turn them into this format with a pipe in-between (IPs made up) | 111.245.48....
uselesslinuxman's user avatar
15 votes
5 answers

What's the best way to take a segment out of a text file?

What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
Chris Huang-Leaver's user avatar
120 votes
13 answers

Replace environment variables in a file with their actual values?

Is there an easy way to substitute/evaluate environment variables in a file? Like let's say I have a file config.xml that contains: <property> <name>instanceId</name> <...
Robert Fraser's user avatar
59 votes
11 answers

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...
user1228191's user avatar
33 votes
5 answers

Multiline pattern match using sed, awk or grep [duplicate]

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. {........
user avatar
569 votes
8 answers

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
Cory Klein's user avatar
  • 18.6k
33 votes
12 answers

Only remove commas embedded within quotes in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
mtk's user avatar
  • 27.2k
21 votes
4 answers

compare two columns of different files and print if it matches

I am using Solaris 10 and so grep options involving -f don't work. I have two pipe-separated files: file1: abc|123|BNY|apple| cab|234|cyx|orange| def|kumar|pki|bird| file 2: abc|123| kumar|pki| ...
user68365's user avatar
  • 231
41 votes
8 answers

How to grep-inverse-match and exclude "before" and "after" lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except (...
Amelio Vazquez-Reina's user avatar
19 votes
7 answers

How can I prepend a tag to the beginning of several files?

I need to add PHP tags surrounding a file. It's easy to append them using find . -exec echo "?>" >> '{}' \; but how can I prepend the tag <?php?
user avatar
10 votes
3 answers

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...
ayrton_senna's user avatar
  • 1,061
98 votes
11 answers

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5
Özzesh's user avatar
  • 3,609
45 votes
7 answers

Non-greedy match with SED regex (emulate perl's .*?)

I want to use sed to replace anything in a string between the first AB and the first occurrence of AC (inclusive) with XXX. For example, I have this string (this string is for a test only): ...
Baba's user avatar
  • 3,229
43 votes
9 answers

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
LanceBaynes's user avatar
  • 39.7k
15 votes
4 answers

sed - Print lines matched by a pattern range if one line matches a condition [duplicate]

I have a multi line log entry format that I need to process. The log looks something like this: --START-- Device=B Data=asdfasdf Lorem=Ipsum --END-- --START-- Device=A Data=asdfasdf Lorem=Ipsum --...
Brad's user avatar
  • 153
12 votes
6 answers

Search for a string and print everything before and after within a range

I have this file: sometext1{ string1 } sometext2{ string2 string3 } sometext3{ string4 string5 string6 } I want to search this file for a specific string and print everything before this string up ...
rodrigo's user avatar
  • 123
11 votes
9 answers

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
Bob's user avatar
  • 211
206 votes
5 answers

How can I wrap text at a certain column size?

I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
cwd's user avatar
  • 44.9k
202 votes
6 answers

Return only the portion of a line after a matching pattern

So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
MaQleod's user avatar
  • 2,614
186 votes
7 answers

How do I delete the first n lines of an ascii file using shell commands?

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so ...
Paul's user avatar
  • 9,263
47 votes
2 answers

Common lines between two files [duplicate]

I have the following code that I run on my Terminal. LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed This doesn't give me the common lines between the two files. ...
Marwah Soliman's user avatar
25 votes
3 answers

Substitute pattern within a file with the content of other file

I have a text file (devel.xml). I added the word REPLACETHIS to it in order to replace this string with the content within a different file (temp.txt). The closest thing I have is this: sed -i -e "/...
RafaelGP's user avatar
  • 371
18 votes
10 answers

Remove extra header lines from file, except for the first line

I have a file that looks like this toy example. My actual file has 4 million lines, about 10 of which I need to delete. ID Data1 Data2 1 100 100 2 100 200 3 200 100 ID Data1 ...
Gaius Augustus's user avatar
10 votes
4 answers

How to print lines number 15 and 25 out of each 50 lines?

I have a large file and I would like print from each sequential 50 lines , the 15th and 25th lines. sed -n '15,25p' inputfile How to modify this command to print only lines 15 and 25 and to loop ...
Mohsen El-Tahawy's user avatar

2 3 4 5