Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
8,359
questions
955
votes
10
answers
1.8m
views
How can I replace a string in a file(s)?
Replacing strings in files based on certain search criteria is a very common task. How can I
replace string foo with bar in all files in the current directory?
do the same recursively for sub ...
- 238k
696
votes
4
answers
310k
views
Why is printf better than echo?
I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
- 12.3k
621
votes
11
answers
1.3m
views
How to append multiple lines to a file
I am writing a bash script to look for a file if it doesn't exist then create it and append this to it:
Host localhost
ForwardAgent yes
So "line then new line 'tab' then text" I think its a ...
- 6,655
569
votes
8
answers
664k
views
Can grep output only specified groupings that match?
Say I have a file:
# file: 'test.txt'
foobar bash 1
bash
foobar happy
foobar
I only want to know what words appear after "foobar", so I can use this regex:
"foobar \(\w\+\)"
The parenthesis ...
- 18.6k
448
votes
2
answers
1.9m
views
Using 'sed' to find and replace [duplicate]
I know this question has probably been answered before. I have seen many threads about this in various places, but the answers are usually hard to extract for me. I am looking for help with an ...
- 4,641
367
votes
21
answers
733k
views
How do I trim leading and trailing whitespace from each line of some output?
I would like to remove all leading and trailing spaces and tabs from each line in an output.
Is there a simple tool like trim I could pipe my output into?
Example file:
test space at back
test ...
- 28.2k
279
votes
18
answers
301k
views
How do you sort du output by size?
How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong.
[~]# du -sh /var/* | sort -n
0 /var/mail
1.2M /...
- 58.5k
278
votes
19
answers
450k
views
How to add a newline to the end of a file?
Using version control systems I get annoyed at the noise when the diff says No newline at end of file.
So I was wondering: How to add a newline at the end of a file to get rid of those messages?
- 15.2k
271
votes
5
answers
70k
views
Why is using a shell loop to process text considered bad practice?
Is using a while loop to process text generally considered bad practice in POSIX shells?
As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
- 152k
230
votes
9
answers
303k
views
Looping through files with spaces in the names? [duplicate]
I wrote the following script to diff the outputs of two directores with all the same files in them as such:
#!/bin/bash
for file in `find . -name "*.csv"`
do
echo "file = $file";
diff $...
- 7,143
220
votes
10
answers
298k
views
How to remove duplicate lines inside a text file?
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).
What I need is to remove all the repetitions ...
- 17.5k
220
votes
6
answers
620k
views
remove particular characters from a variable using bash
I want to parse a variable (in my case it's development kit version) to make it dot(.) free. If version='2.3.3', desired output is 233.
I tried as below, but it requires . to be replaced with another ...
- 4,817
206
votes
5
answers
114k
views
How can I wrap text at a certain column size?
I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
- 44.9k
204
votes
9
answers
722k
views
Split string by delimiter and get N-th element
I have a string:
one_two_three_four_five
I need to save in a variable A value two and in variable B value fourfrom the above string
I am using ksh.
- 2,295
202
votes
6
answers
696k
views
Return only the portion of a line after a matching pattern
So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
- 2,614
186
votes
7
answers
426k
views
How do I delete the first n lines of an ascii file using shell commands?
I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so ...
- 9,263
181
votes
10
answers
160k
views
How do I use cut to separate by multiple whitespace?
I have this input, which is displayed in columns. I would like to get the second last column with the numbers of this sample:
[ 3] 1.0- 2.0 sec 1.00 MBytes 8.39 Mbits/sec
[ 3] 2.0- 3.0 sec 768 ...
- 28.2k
179
votes
6
answers
511k
views
With the Linux "cat" command, how do I show only certain lines by number
If I use cat -n text.txt to automatically number the lines, how do I then use the command to show only certain numbered lines.
- 1,899
166
votes
20
answers
84k
views
A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1
Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ?
I have my ...
- 32.7k
162
votes
13
answers
98k
views
How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB disk space?
How do I remove the first 300 million lines from a 700 GB text file
on a system with 1 TB disk space total, with 300 GB available?
(My system has 2 GB of memory.)
The answers I found use sed, tail, ...
- 1,273
160
votes
6
answers
398k
views
How to merge all (text) files in a directory into one?
I've got 14 files all being parts of one text. I'd like to merge them into one. How to do that?
- 17.5k
160
votes
16
answers
440k
views
How to insert text before the first line of a file?
I've been looking around sed command to add text into a file in a specific line.
This works adding text after line 1:
sed '1 a\
But I want to add it before line 1. It would be:
sed '0 a\
but I get ...
- 1,737
155
votes
19
answers
162k
views
Decoding URL encoding (percent encoding)
I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this?
I did search a bit through unix.stackexchange.com and on the ...
- 11.5k
147
votes
13
answers
288k
views
Replace space with new line
How can I replace spaces with new lines on an input like:
/path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5 etc...
To obtain the following:
/path/to/file
/path/to/file2
/...
- 4,339
138
votes
23
answers
185k
views
Is there a way to get the min, max, median, and average of a list of numbers in a single command?
I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script.
Although my immediate situation is for ...
- 32.7k
138
votes
4
answers
503k
views
How to insert text after a certain string in a file?
Right now I'm using
echo "Hello World" >> file.txt
to append some text to a file but I also need to add text below a certain string let's say [option], is it possible with sed?
EG:
...
- 1,687
134
votes
20
answers
343k
views
How to count the number of a specific character in each line?
I was wondering how to count the number of a specific character in each line by some text processing utilities?
For example, to count " in each line of the following text
"hello!"
Thank you!
The ...
- 100k
133
votes
14
answers
33k
views
Replace string in a huge (70GB), one line, text file
I have a huge (70GB), one line, text file and I want to replace a string (token) in it.
I want to replace the token <unk>, with another dummy token (glove issue).
I tried sed:
sed 's/<unk&...
- 1,457
123
votes
8
answers
229k
views
Grabbing the first [x] characters for a string from a pipe
If I have really long output from a command (single line) but I know I only want the first [x] (let's say 8) characters of the output, what's the easiest way to get that? There aren't any delimiters.
- 58.5k
121
votes
7
answers
214k
views
Convert file contents to lower case
I have temp file with some lower-case and upper-case contents.
Input
Contents of my temp file:
hi
Jigar
GANDHI
jiga
I want to convert all upper to lower.
Command
I tried the following command:
...
- 4,860
120
votes
13
answers
160k
views
Replace environment variables in a file with their actual values?
Is there an easy way to substitute/evaluate environment variables in a file?
Like let's say I have a file config.xml that contains:
<property>
<name>instanceId</name>
<...
- 1,307
119
votes
6
answers
299k
views
How to do an if statement from the result of an executed command
I am trying to do an IF statement from the output of an executed commmand. Here is how I am trying to do it, but it doesn't work. Does anyone know the right way to do this?
if [ "`netstat -lnp ...
- 4,087
117
votes
4
answers
228k
views
How to print only the duplicate values from a text file?
Suppose there is a column of numeric values like following:
File1:
1
2
3
3
3
4
4
4
5
6
I want the output:
3
4
That is, only the repeated lines. Are there any command line tools to find this ...
- 2,149
117
votes
4
answers
52k
views
Does sort support sorting a file in-place, like `sed --in-place`?
Am I blind or is there no option like --in-place for sort?
In order to save results to the input file, sed uses -i (--in-place).
Redirecting the output of sort to the input file
sort < f > f
...
112
votes
8
answers
98k
views
Show all the file up to the match
grep --before-context 5 shows 5 lines before the match.
I want to show everything before the match.
Doing grep --before-context 99999999 would work but it is not very... professional.
How to show ...
- 8,095
110
votes
5
answers
177k
views
How do I remove the newline from the last line in a file in order to add text to that line?
Suppose I have a file called file:
$ cat file
Hello
Welcome to
Unix
I want to add and Linux at the end of the last line of the file. If I do echo " and Linux" >> file will be added ...
- 24.2k
107
votes
7
answers
359k
views
How to strip multiple spaces to one using sed?
sed on AIX is not doing what I think it should.
I'm trying to replace multiple spaces with a single space in the output of IOSTAT:
# iostat
System configuration: lcpu=4 drives=8 paths=2 vdisks=0
tty:...
- 1,314
104
votes
4
answers
138k
views
Why is `while IFS= read` used so often, instead of `IFS=; while read..`?
It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
- 32.7k
102
votes
16
answers
81k
views
sort but keep header line at the top
I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
- 2,059
102
votes
5
answers
247k
views
Remove line containing certain string and the following line
I use this
cat foo.txt | sed '/bar/d'
to remove lines containing the string bar in the file.
I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
- 3,173
99
votes
11
answers
194k
views
How to test whether a file uses CRLF or LF without modifying it?
I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and ...
- 5,531
98
votes
11
answers
353k
views
Remove last character from line
I want to remove last character from a line:
[root@ozzesh ~]#df -h | awk '{ print $5 }'
Use%
22%
1%
1%
59%
51%
63%
5%
Expected result:
Use
22
1
1
59
51
63
5
- 3,609
96
votes
15
answers
78k
views
Removing control chars (including console codes / colours) from script output
I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
- 1,071
96
votes
4
answers
57k
views
How can I use column to delimit on tabs and not spaces?
I'd like to use the Unix column command to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I ...
- 2,637
96
votes
1
answer
86k
views
How to truncate text lines to N characters maximum? [duplicate]
Given a text file, or the output of a command, how can I truncate it so that every line longer than N characters (usually N=80 in a terminal) gets shorten to N characters maximum?
- 19.6k
95
votes
9
answers
30k
views
Is there a tool that combines zcat and cat transparently?
When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this:
$ zcat *
you end up with a command line like zcat xyz.log xyz.log.1 xyz....
- 16.4k
94
votes
13
answers
28k
views
Linux tools to treat files as sets and perform set operations on them
Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
- 1,043
91
votes
12
answers
174k
views
How to print all lines after a match up to the end of the file?
Input file1 is:
dog 123 4335
cat 13123 23424
deer 2131 213132
bear 2313 21313
I give the match the pattern from in other file ( like dog 123 4335 from file2).
I match the pattern of the line is ...
- 1,109
91
votes
7
answers
73k
views
How can I make iconv replace the input file with the converted output?
I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT.
Since adding the -o parameter ( in my experience ) actually writes ...
- 1,055
89
votes
13
answers
519k
views
How to parse JSON with shell scripting in Linux?
I have a JSON output from which I need to extract a few parameters in Linux.
This is the JSON output:
{
"OwnerId": "121456789127",
"ReservationId": "r-48465168",
"Groups": [],...
- 991