Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

81
votes
3answers
9k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
317
votes
5answers
121k views

Why is printf better than echo?

I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
281
votes
6answers
551k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
55
votes
4answers
39k views

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
73
votes
10answers
86k views

Looping through files with spaces in the names?

I wrote the following script to diff the outputs of two directores with all the same files in them as such: #!/bin/bash for file in `find . -name "*.csv"` do echo "file = $file"; diff $...
32
votes
6answers
21k views

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
37
votes
5answers
24k views

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
35
votes
9answers
10k views

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
34
votes
13answers
16k views

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
95
votes
13answers
66k views

how do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M /...
8
votes
8answers
19k views

Transposing rows and columns

I have a file with the lines like below. title1:A1 title2:A2 title3:A3 title4:A4 title5:A5 title1:B1 title2:B2 title3:B3 title4:B4 title5:B5 title1:C1 title2:C2 title3:C3 title4:C4 title5:C5 ...
68
votes
6answers
62k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
49
votes
9answers
12k views

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
10
votes
5answers
1k views

What's the best way to take a segment out of a text file?

What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
41
votes
14answers
16k views

A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1

Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ? I have my ...
6
votes
4answers
618 views

Search for a string and print everything before and after within a range

I have this file: sometext1{ string1 } sometext2{ string2 string3 } sometext3{ string4 string5 string6 } I want to search this file for a specific string and print everything before this string ...
8
votes
2answers
3k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
16
votes
7answers
26k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
15
votes
7answers
2k views

How can I prepend a tag to the beginning of several files?

I need to add PHP tags surrounding a file. It's easy to append them using find . -exec echo "?>" >> '{}' \; but how can I prepend the tag <?php?
11
votes
5answers
3k views

Why does shell Command Substitution gobble up a trailing newline char?

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p # ...
5
votes
3answers
3k views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
25
votes
4answers
21k views

How to get only the unique results without having to sort data?

$ cat data.txt aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $ The result that I need is to display ...
21
votes
1answer
2k views

How to implement a horizontal cat? [duplicate]

Standard cat concatenates files line by line (row by row, if you will). I find myself needing a horizontal cat command more and more often recently; i.e. a command that takes a list of files and ...
16
votes
9answers
1k views

Filter file by line number

Given a file L with one non-negative integer per line and text file F, what would be a fast way to keep only those lines in F, whose line number appears in file L? Example: $ cat L.txt 1 3 $ cat F....
7
votes
1answer
4k views

How to grep all lines where starting line ends with a pattern and ending line ends with another?

I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work. Input: kkkkkkkkkkk ...
17
votes
9answers
15k views

How can I print lines from file backwards (without using “tac”)?

I want to print lines from a file backwards without using tac command. Is there any other solution to do such thing with bash?
6
votes
2answers
10k views

Show only text between 2 matching pattern

I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed If param1 is line matching pattern from other file using sed, below command will works fine:...
101
votes
5answers
78k views

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
15
votes
4answers
32k views

Text between two tags

I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...
14
votes
2answers
28k views

How to cut (select) a field from text line counting from the end?

I know how to select a field from a line using the cut command. For instance, given the following data: a,b,c,d,e f,g,h,i,j k,l,m,n,o This command: cut -d, -f2 # returns the second field of the ...
7
votes
6answers
16k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are ...
10
votes
6answers
14k views

How to grep for text in a file and display the paragraph that has the text?

Below is the text in the file: Pseudo name=Apple Code=42B state=fault Pseudo name=Prance Code=43B state=good I need to grep for "42B" and get the output from the above text like: Pseudo name=...
8
votes
4answers
30k views

compare two columns of different files and print if it matches

I am using solaris 10 and Grep options involving -f doesnt work . I have two pipe separated files: file1: abc|123|BNY|apple| cab|234|cyx|orange| def|kumar|pki|bird| file 2 : abc|123| kumar|pki|...
8
votes
7answers
21k views

Print odd-numbered lines, print even-numbered lines

I want to print the odd-numbered and even-numbered lines from files. I found this shell script which makes use of echo. #!/bin/bash # Write a shell script that, given a file name as the argument ...
3
votes
4answers
467 views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...
21
votes
9answers
41k views

Command to display first few and last few lines of a file

I have a file with many rows, and each row has a timestamp at the starting, like [Thread-3] (21/09/12 06:17:38:672) logged message from code..... So, I frequently check 2 things from this log file. ...
12
votes
3answers
2k views

How can I redirect matching lines to a file, and non-matching lines to a different file?

Is there a script/program/utility already available for the following requirement in a optimised way? someCommand | tee >(grep "pattern" > LinesWhichMatch) | grep -v "pattern" > ...
12
votes
1answer
904 views

How to do `head` and `tail` on null-delimited input in bash?

find command can output names of files as a null-delimited strings (if -print0 is provided), and xargs can consume them with -0 option turned on. But in between, it's hard to manipulate that ...
4
votes
4answers
4k views

Convert output of tree command to json format

Is there a convenient way to convert the output of the *nix command "tree" to JSON format? Edit: I think I didn't describe my problem well enough. My goal is to convert something like: . |-- dir1 | ...
13
votes
7answers
15k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...
6
votes
3answers
451 views

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...
8
votes
4answers
38k views

Shell script to send an alert mail if disk usage exceeds 90%

I am new to shell scripting and want to implement a script on my server which will automatically send email alerts if the disk usage goes over 90% and second email alert when it goes over 95%. My ...
6
votes
1answer
368 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
5
votes
7answers
2k views

Turn list into single line with delimiter

I have to take a list (loads) of IP addresses in this format: 134.27.128.0 111.245.48.0 109.21.244.0 and turn them into this format with a pipe in-between (IPs made up) 134.27.128.0 | 111.245....
3
votes
2answers
4k views

Find the line number which contains the pattern using custom regex delimiter

As in example I'm trying to get line numbers which contains the pattern. My pattern contains slashes so I wanted to add custom delimiter. This simple one works: sed -n '/file/=' temp.txt Using ...
3
votes
6answers
198 views

Find a pattern and insert # at the beginning of 2 line before that and 1 line after that

I have a big file (more than 2000 lines). Where I have to insert a # at the beginning of 2 above lines and at the beginning of 1 below line after finding a pattern. Also insert # at the beginning of ...
2
votes
2answers
138 views

match first column of file a with paragraphs of file b

I have 2 files. The first, fileA looks like TCONS_00000066 XLOC_000030 - u q1:XLOC_000030|TCONS_00000066|0|0.000000|0.000000|0.000000|0.000000|- TCONS_00000130 XLOC_000057 - ...
7
votes
1answer
5k views

How to merge two files in corresponding row?

Now,I have two files: aaaa.txt: a=0; b=1; c=2; bbbb.txt: d=3 e=4 f=5 I want to merge aaaa.txt and bbbb.txt to cccc.txt. cccc.txt as follow: a=0;d=3 b=1;e=4 c=2;f=5 So, what can I do for ...
6
votes
5answers
1k views

What command(s) will feed a tab-delimited text file and cut each line to 80 characters?

I've got multiple-line text files of (sometimes) tab-delimited data. I'd like to output the file so I can glance over it - so I'd like to only see the first 80 characters of each line (I designed the ...
6
votes
4answers
7k views

Compare two files: lines present in one, not in the other, by one column comparison

I need to compare 2 files. Column 1 is the same in both files. Column 2 is what I want to compare: I want all lines in file 1 that are not in file 2 when comparing column 2. Column 3 is different in ...