Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

Filter by
Sorted by
Tagged with
955 votes
10 answers

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
terdon's user avatar
  • 238k
696 votes
4 answers

Why is printf better than echo?

I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
amphibient's user avatar
  • 12.3k
621 votes
11 answers

How to append multiple lines to a file

I am writing a bash script to look for a file if it doesn't exist then create it and append this to it: Host localhost ForwardAgent yes So "line then new line 'tab' then text" I think its a ...
TheLegend's user avatar
  • 6,655
569 votes
8 answers

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
Cory Klein's user avatar
  • 18.6k
448 votes
2 answers

Using 'sed' to find and replace [duplicate]

I know this question has probably been answered before. I have seen many threads about this in various places, but the answers are usually hard to extract for me. I am looking for help with an ...
roo's user avatar
  • 4,641
367 votes
21 answers

How do I trim leading and trailing whitespace from each line of some output?

I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into? Example file: test space at back test ...
rubo77's user avatar
  • 28.2k
279 votes
18 answers

How do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M /...
xenoterracide's user avatar
278 votes
19 answers

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid of those messages?
k0pernikus's user avatar
  • 15.2k
271 votes
5 answers

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
cuonglm's user avatar
  • 152k
230 votes
9 answers

Looping through files with spaces in the names? [duplicate]

I wrote the following script to diff the outputs of two directores with all the same files in them as such: #!/bin/bash for file in `find . -name "*.csv"` do echo "file = $file"; diff $...
Amir Afghani's user avatar
  • 7,143
220 votes
10 answers

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
Ivan's user avatar
  • 17.5k
220 votes
6 answers

remove particular characters from a variable using bash

I want to parse a variable (in my case it's development kit version) to make it dot(.) free. If version='2.3.3', desired output is 233. I tried as below, but it requires . to be replaced with another ...
prayagupa's user avatar
  • 4,817
206 votes
5 answers

How can I wrap text at a certain column size?

I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
cwd's user avatar
  • 44.9k
204 votes
9 answers

Split string by delimiter and get N-th element

I have a string: one_two_three_four_five I need to save in a variable A value two and in variable B value fourfrom the above string I am using ksh.
Alex's user avatar
  • 2,295
202 votes
6 answers

Return only the portion of a line after a matching pattern

So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
MaQleod's user avatar
  • 2,614
186 votes
7 answers

How do I delete the first n lines of an ascii file using shell commands?

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so ...
Paul's user avatar
  • 9,263
181 votes
10 answers

How do I use cut to separate by multiple whitespace?

I have this input, which is displayed in columns. I would like to get the second last column with the numbers of this sample: [ 3] 1.0- 2.0 sec 1.00 MBytes 8.39 Mbits/sec [ 3] 2.0- 3.0 sec 768 ...
rubo77's user avatar
  • 28.2k
179 votes
6 answers

With the Linux "cat" command, how do I show only certain lines by number

If I use cat -n text.txt to automatically number the lines, how do I then use the command to show only certain numbered lines.
Geckono1's user avatar
  • 1,899
166 votes
20 answers

A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1

Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ? I have my ...
Peter.O's user avatar
  • 32.7k
162 votes
13 answers

How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB disk space?

How do I remove the first 300 million lines from a 700 GB text file on a system with 1 TB disk space total, with 300 GB available?  (My system has 2 GB of memory.)  The answers I found use sed, tail, ...
Kris's user avatar
  • 1,273
160 votes
6 answers

How to merge all (text) files in a directory into one?

I've got 14 files all being parts of one text. I'd like to merge them into one. How to do that?
Ivan's user avatar
  • 17.5k
160 votes
16 answers

How to insert text before the first line of a file?

I've been looking around sed command to add text into a file in a specific line. This works adding text after line 1: sed '1 a\ But I want to add it before line 1. It would be: sed '0 a\ but I get ...
Manolo's user avatar
  • 1,737
155 votes
19 answers

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through and on the ...
DisplayName's user avatar
  • 11.5k
147 votes
13 answers

Replace space with new line

How can I replace spaces with new lines on an input like: /path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5 etc... To obtain the following: /path/to/file /path/to/file2 /...
laconbass's user avatar
  • 4,339
138 votes
23 answers

Is there a way to get the min, max, median, and average of a list of numbers in a single command?

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script. Although my immediate situation is for ...
Peter.O's user avatar
  • 32.7k
138 votes
4 answers

How to insert text after a certain string in a file?

Right now I'm using echo "Hello World" >> file.txt to append some text to a file but I also need to add text below a certain string let's say [option], is it possible with sed? EG: ...
Javier Villanueva's user avatar
134 votes
20 answers

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
Tim's user avatar
  • 100k
133 votes
14 answers

Replace string in a huge (70GB), one line, text file

I have a huge (70GB), one line, text file and I want to replace a string (token) in it. I want to replace the token <unk>, with another dummy token (glove issue). I tried sed: sed 's/<unk&...
Christos Baziotis's user avatar
123 votes
8 answers

Grabbing the first [x] characters for a string from a pipe

If I have really long output from a command (single line) but I know I only want the first [x] (let's say 8) characters of the output, what's the easiest way to get that? There aren't any delimiters.
xenoterracide's user avatar
121 votes
7 answers

Convert file contents to lower case

I have temp file with some lower-case and upper-case contents. Input Contents of my temp file: hi Jigar GANDHI jiga I want to convert all upper to lower. Command I tried the following command: ...
JigarGandhi's user avatar
  • 4,860
120 votes
13 answers

Replace environment variables in a file with their actual values?

Is there an easy way to substitute/evaluate environment variables in a file? Like let's say I have a file config.xml that contains: <property> <name>instanceId</name> <...
Robert Fraser's user avatar
119 votes
6 answers

How to do an if statement from the result of an executed command

I am trying to do an IF statement from the output of an executed commmand. Here is how I am trying to do it, but it doesn't work. Does anyone know the right way to do this? if [ "`netstat -lnp ...
djangofan's user avatar
  • 4,087
117 votes
4 answers

How to print only the duplicate values from a text file?

Suppose there is a column of numeric values like following: File1: 1 2 3 3 3 4 4 4 5 6 I want the output: 3 4 That is, only the repeated lines. Are there any command line tools to find this ...
N. F.'s user avatar
  • 2,149
117 votes
4 answers

Does sort support sorting a file in-place, like `sed --in-place`?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
Grzegorz Wierzowiecki's user avatar
112 votes
8 answers

Show all the file up to the match

grep --before-context 5 shows 5 lines before the match. I want to show everything before the match. Doing grep --before-context 99999999 would work but it is not very... professional. How to show ...
Nicolas Raoul's user avatar
110 votes
5 answers

How do I remove the newline from the last line in a file in order to add text to that line?

Suppose I have a file called file: $ cat file Hello Welcome to Unix I want to add and Linux at the end of the last line of the file. If I do echo " and Linux" >> file will be added ...
Pandya's user avatar
  • 24.2k
107 votes
7 answers

How to strip multiple spaces to one using sed?

sed on AIX is not doing what I think it should. I'm trying to replace multiple spaces with a single space in the output of IOSTAT: # iostat System configuration: lcpu=4 drives=8 paths=2 vdisks=0 tty:...
WernerCD's user avatar
  • 1,314
104 votes
4 answers

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
Peter.O's user avatar
  • 32.7k
102 votes
16 answers

sort but keep header line at the top

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
jonderry's user avatar
  • 2,059
102 votes
5 answers

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
jakub.g's user avatar
  • 3,173
99 votes
11 answers

How to test whether a file uses CRLF or LF without modifying it?

I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and ...
Adam Ryczkowski's user avatar
98 votes
11 answers

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5
Özzesh's user avatar
  • 3,609
96 votes
15 answers

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
andrew cooke's user avatar
  • 1,071
96 votes
4 answers

How can I use column to delimit on tabs and not spaces?

I'd like to use the Unix column command to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I ...
turtle's user avatar
  • 2,637
96 votes
1 answer

How to truncate text lines to N characters maximum? [duplicate]

Given a text file, or the output of a command, how can I truncate it so that every line longer than N characters (usually N=80 in a terminal) gets shorten to N characters maximum?
Totor's user avatar
  • 19.6k
95 votes
9 answers

Is there a tool that combines zcat and cat transparently?

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this: $ zcat * you end up with a command line like zcat xyz.log xyz.log.1 xyz....
0xC0000022L's user avatar
  • 16.4k
94 votes
13 answers

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
nilton's user avatar
  • 1,043
91 votes
12 answers

How to print all lines after a match up to the end of the file?

Input file1 is: dog 123 4335 cat 13123 23424 deer 2131 213132 bear 2313 21313 I give the match the pattern from in other file ( like dog 123 4335 from file2). I match the pattern of the line is ...
loganaayahee's user avatar
  • 1,109
91 votes
7 answers

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
meder omuraliev's user avatar
89 votes
13 answers

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": [],...
user3086014's user avatar

2 3 4 5