306
votes
Accepted
Split string by delimiter and get N-th element
Use cut with _ as the field delimiter and get desired fields:
A="$(cut -d'_' -f2 <<<'one_two_three_four_five')"
B="$(cut -d'_' -f4 <<<'one_two_three_four_five')"
You can also use ...
- 56.9k
291
votes
Accepted
How do I use cut to separate by multiple whitespace?
If we use tr command along with squeeze option (-s flag ) to convert all multiple consecutive spaces to a single space and then perform cut operation with space as delimiter – we can access the ...
- 2,926
171
votes
Accepted
Replace environment variables in a file with their actual values?
You could use envsubst (part of gnu gettext):
envsubst < infile
will replace the environment variables in your file with their corresponding value. The variable names must consist solely of ...
- 83.9k
159
votes
How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB disk space?
Removing the first n lines (or bytes) can be done in-place using dd (or alternatively using loop devices). It does not use a temporary file and there is no size limit; however, it is dangerous since ...
- 49.9k
133
votes
Accepted
How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB disk space?
If you have enough space to compress the file, which should free a significant amount of space allowing you to do other operations, you can try this:
gzip file && zcat file.gz | tail -n +...
- 245k
118
votes
Accepted
Counting occurrences of word in text file
Given such a requirement, I would use a GNU grep (for the -o option), then pass it through wc to count the total number of occurrences:
$ grep -o -i iphone Tweet_Data | wc -l
3
Plain grep -c on the ...
- 67.8k
116
votes
Replace string in a huge (70GB), one line, text file
For such a big file, one possibility is Flex. Let unk.l be:
%%
\<unk\> printf("<raw_unk>");
%%
Then compile and execute:
$ flex -o unk.c unk.l
$ cc -o unk -O2 unk.c -lfl
$ unk &...
- 12.4k
112
votes
Printing every Nth line out of a large file into a new file
awk 'NR % 5 == 0' input > output
This prints every fifth line.
To use an environment variable:
NUM=5
awk -v NUM=$NUM 'NR % NUM == 0' input > output
- 2,626
111
votes
Accepted
Replace string in a huge (70GB), one line, text file
The usual text processing tools are not designed to handle lines that don't fit in RAM. They tend to work by reading one record (one line), manipulating it, and outputting the result, then proceeding ...
104
votes
Accepted
Common lines between two files
Use comm -12 file1 file2 to get common lines in both files.
You may also needs your file to be sorted to comm to work as expected.
comm -12 <(sort file1) <(sort file2)
From man comm:
-1 ...
- 41.6k
101
votes
How to test whether a file uses CRLF or LF without modifying it?
Note: This should not be used in automation scripts, just for checking quickly. For automation scripts, I'd suggest you look at other answers
Use cat -A
$ cat file
hello
hello
Now if this file was ...
- 4,189
94
votes
Accepted
Remove last character from string captured with awk
Yes, with substr() you can do string slicing:
... | awk '{if (NR!=1) {print substr($2, 1, length($2)-1)}}'
length($2) will get us the length of the second field, deducting 1 from that to strip off ...
- 56.9k
93
votes
Accepted
How to cut till first delimiter and get remaining part of strings?
Simply with cut command:
echo "pandi/sha/Dev/bin/boot" | cut -d'/' -f2-
sha/Dev/bin/boot
-d'/' - field delimiter
-f2- - a range of fields to output (-f<from>-<to> ; in our case: from 2 ...
- 30.5k
79
votes
Accepted
How do you keep only the last n lines of a log file?
It is possible like this, but as others have said, the safest option is the generation of a new file and then a move of that file to overwrite the original.
The below method loads the lines into BASH,...
- 1,217
66
votes
How do I remove the newline from the last line in a file in order to add text to that line?
You can achieve this with perl as:
perl -pi -e 'chomp if eof' myfile
Compared to truncate or dd this not gonna leave you with a broken file if myfile had actually no trailing newlines.
(the answer ...
- 5,212
66
votes
Merging contents of multiple .csv files into single .csv file
The simplest approach for achieving that would be typing the following command
cat *csv > combined.csv
This file would contain the contents of all your csv files just in the way you mentioned.
- 867
65
votes
Accepted
How to count the times a specific character appears in a file?
You can combine tr (translate or delete characters) with wc (count words, lines, characters):
tr -cd '"' < yourfile.cfg | wc -c
-delete all characters in the complement of ", and then ...
- 16.2k
62
votes
Accepted
What is the quickest way of replacing 0 by 1 and vice-versa in a stream?
You can use tr for this, its main purpose is character translation:
echo 111111100000000000000 | tr 01 10
Your sed command replaces all 0s with 1s, resulting in a string containing only 1s (the ...
- 450k
58
votes
Split string by delimiter and get N-th element
Wanted to see an awk answer, so here's one:
A=$(awk -F_ '{print $2}' <<< 'one_two_three_four_five')
B=$(awk -F_ '{print $4}' <<< 'one_two_three_four_five')
Try it online!
- 938
56
votes
How to compare two different files line by line in unix?
Use diff command as following, in bash or any other shell that supports <(...) process substitutions or you can emulate it as shown here:
diff --new-line-format='FALSE'$'\n' \
--old-line-...
- 41.6k
55
votes
How do I delete the first n lines of an ascii file using shell commands?
sed -i '1,3d' file.txt
This deletes first 3 line from file.txt.
- 1,321
55
votes
Replace environment variables in a file with their actual values?
This is not very nice but it works
( echo "cat <<EOF" ; cat config.xml ; echo EOF ) | sh
If it was in a shell script it would look like:
#! /bin/sh
cat <<EOF
<property>
<...
- 2,930
54
votes
Accepted
Converting CSV to TSV
Python
Add to file named csv2tab, and make it executable
touch csv2tab && chmod u+x csv2tab
Add to it
#!/usr/bin/env python
import csv, sys
csv.writer(sys.stdout, dialect='excel-tab')....
- 1,229
53
votes
Can grep output only specified groupings that match?
pcregrep has a smarter -o option
that lets you choose which capturing groups you want output.
So, using your example file,
$ pcregrep -o1 "foobar (\w+)" test.txt
bash
happy
53
votes
How to remove a column or multiple columns from file using shell command?
If you are content to clear the columns rather than completely removing them:
Don't print 6th and 8th column
awk '{$6=$8=""; print $0}' file
- 27.9k
53
votes
Printing every Nth line out of a large file into a new file
To print every N th line, usesed -n '0~Np'
For example, to copy every 5th line of oldfile to newfile, do
sed -n '0~5p' oldfile > newfile
This uses sed’s first ~step address form,
which means “...
50
votes
Accepted
Splitting a file into multiple files based on 1st column value
Try:
awk -F\| '{print>$1}' file1
This writes each line to a file named after the first column.
How it works:
-F\| sets the field separator to |.
print>$1 prints the current line to a file ...
- 75.4k
50
votes
append text with echo without new line
Assuming that the file does not already end in a newline and you simply want to append some more text without adding one, you can use the -n argument, e.g.
echo -n "some text here" >> file.txt
...
- 995
49
votes
With the Linux "cat" command, how do I show only certain lines by number
You can use awk straight up.
awk 'NR==1' file.txt
replacing '1' with the desired line number.
- 591
Only top scored, non community-wiki answers of a minimum length are eligible