Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

276
votes
5answers
101k views

Why is printf better than echo?

I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
65
votes
3answers
6k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
183
votes
5answers
374k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
54
votes
4answers
32k views

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
58
votes
10answers
74k views

Looping through files with spaces in the names?

I wrote the following script to diff the outputs of two directores with all the same files in them as such: #!/bin/bash for file in `find . -name "*.csv"` do echo "file = $file"; diff ...
27
votes
6answers
18k views

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
32
votes
4answers
20k views

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
34
votes
13answers
15k views

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
33
votes
8answers
9k views

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
78
votes
13answers
51k views

how do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M ...
10
votes
5answers
1k views

What's the best way to take a segment out of a text file?

What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
61
votes
6answers
47k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
43
votes
9answers
10k views

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
7
votes
6answers
16k views

Transposing rows and columns

I have a file with the lines like below. title1:A1 title2:A2 title3:A3 title4:A4 title5:A5 title1:B1 title2:B2 title3:B3 title4:B4 title5:B5 title1:C1 title2:C2 title3:C3 title4:C4 title5:C5 ...
34
votes
14answers
12k views

A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1

Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ? I have my ...
7
votes
2answers
2k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
15
votes
7answers
2k views

How can I prepend a tag to the beginning of several files?

I need to add PHP tags surrounding a file. It's easy to append them using find . -exec echo "?>" >> '{}' \; but how can I prepend the tag <?php?
13
votes
7answers
21k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
5
votes
3answers
2k views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
10
votes
6answers
8k views

How do I recursively grep through compressed archives?

I'm trying to find out what modules use Test::Version in cpan. So I've used minicpan to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files ...
13
votes
4answers
26k views

Text between two tags

I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...
11
votes
5answers
3k views

Why does shell Command Substitution gobble up a trailing newline char?

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p # ...
6
votes
2answers
8k views

Show only text between 2 matching pattern

I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed If param1 is line matching pattern from other file using sed, below command will works ...
5
votes
3answers
485 views

Using sed to search for a string and print everything before and after within a range

I have this file: sometext1{ string1 } sometext2{ string2 string3 } sometext3{ string4 string5 string6 } I want to search this file for a specific string and print everything before this string ...
15
votes
9answers
954 views

Filter file by line number

Given a file L with one non-negative integer per line and text file F, what would be a fast way to keep only those lines in F, whose line number appears in file L? Example: $ cat L.txt 1 3 $ cat ...
5
votes
4answers
20k views

compare two columns of different files and print if it matches

I am using solaris 10 and Grep options involving -f doesnt work . I have two pipe separated files: file1: abc|123|BNY|apple| cab|234|cyx|orange| def|kumar|pki|bird| file 2 : abc|123| ...
4
votes
2answers
318 views

how to sort by the day of the week?

I have two files. file1: Dave 734.838.9800 Bob 313.123.4567 Carol 248.344.5576 Mary 313.449.1390 Ted 248.496.2204 Alice 616.556.4458 file2: Bob Tuesday Carol Monday Ted Sunday ...
3
votes
4answers
373 views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...
74
votes
4answers
59k views

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
24
votes
4answers
16k views

How to get only the unique results without having to sort data?

$ cat data.txt aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $ The result that I need is to display ...
20
votes
1answer
2k views

How to implement a horizontal cat? [duplicate]

Standard cat concatenates files line by line (row by row, if you will). I find myself needing a horizontal cat command more and more often recently; i.e. a command that takes a list of files and ...
13
votes
5answers
37k views

How do I append text to the beginning and end of multiple text files in Bash?

I have a directory full of text files. My goal is to append text to the beginning and end of all of them. The text that goes at the beginning and end is the same for each file. Based on code I got ...
12
votes
1answer
750 views

How to do `head` and `tail` on null-delimited input in bash?

find command can output names of files as a null-delimited strings (if -print0 is provided), and xargs can consume them with -0 option turned on. But in between, it's hard to manipulate that ...
18
votes
4answers
35k views

Delete last line from the file

I use sed to quickly delete lines with specific position as sed '1d' sed '5d' But, what if I want to delete the last line of the file and I don't know the count of lines (I know I can get that ...
6
votes
1answer
3k views

How to grep all lines where starting line ends with a pattern and ending line ends with another?

I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work. Input: kkkkkkkkkkk ...
4
votes
4answers
3k views

Convert output of tree command to json format

Is there a convenient way to convert the output of the *nix command "tree" to JSON format? Edit: I think I didn't describe my problem well enough. My goal is to convert something like: . |-- dir1 | ...
8
votes
4answers
30k views

Shell script to send an alert mail if disk usage exceeds 90%

I am new to shell scripting and want to implement a script on my server which will automatically send email alerts if the disk usage goes over 90% and second email alert when it goes over 95%. My ...
4
votes
3answers
5k views

How to bulk convert all the file in a file system branch between Unix and Windows line break format?

Everybody knows :-) that in Windows plain text files lines are terminated with CR+LF, and in Unix&Linux - with LF only. How can I quickly convert all my source code files from one format to ...
3
votes
4answers
558 views

Shell script or command to Strip text and Binary files excerpts from large Log Files

I need to remove a large binary file (PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the ...
1
vote
1answer
44 views

Random place change in row with restrictions

How could I randomly change places of characters in a text file, with some probability (say 10%)? file: 1 34567 2 12356 3 16987 outcome: 1 35674 2 12653 3 61987
14
votes
14answers
5k views

How to read an IP address backwards?

If we have this string (IP address): 192.168.1.1 How can I derive the (DNS reverse record form) from this string, so it will be shown like 1.1.168.192.in-addr.arpa using a shell script?
9
votes
5answers
9k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
20
votes
11answers
18k views

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through unix.stackexchange.com and on the ...
10
votes
2answers
23k views

How to cut (select) a field from text line counting from the end?

I know how to select a field from a line using the cut command. For instance, given the following data: a,b,c,d,e f,g,h,i,j k,l,m,n,o This command: cut -d, -f2 # returns the second field of the ...
34
votes
7answers
4k views

Is there a tool that combines zcat and cat transparently?

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this: $ zcat * you end up with a command line like zcat xyz.log xyz.log.1 ...
9
votes
1answer
4k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
8
votes
6answers
13k views

Concatenate multiple files with same header

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other ...
6
votes
3answers
1k views

Fast way to build a test file with every second listed in YYYY-mm-dd HH:MM:SS format

I want to create a large test file with lines containg dates listed by the second, but my method is taking inordinately long... (or at least, that's how it feels :) ... 43 minutes to create only ...
13
votes
5answers
685 views

non-line-oriented tool for string replacement?

I recently asked a question about how to remove the newline character if it occurs after another specific character. Unix text-processing tools are very powerful, but almost all of them deal with ...
10
votes
5answers
47k views

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5