Manipulation of text by programs, scripts, etc.
17
votes
4answers
5k views
Why is `while IFS= read` used so often, instead of `IFS=; while read..`?
It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
18
votes
6answers
2k views
Is there a way to modify a file in-place?
I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
13
votes
2answers
4k views
How can I make iconv replace the input file with the converted output?
I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT.
Since adding the -o parameter ( in my experience ) actually writes ...
13
votes
8answers
2k views
Is there a robust command line tool for processing csv files?
I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
11
votes
3answers
3k views
Removing control chars (including console codes / colours) from script output
I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
9
votes
6answers
912 views
How can I prepend a tag to the beginning of several files?
I need to add PHP tags surrounding a file. It's easy to append them using
find . -exec echo "?>" >> '{}' \;
but how can I prepend the tag <?php?
21
votes
6answers
2k views
Linux tools to treat files as sets and perform set operations on them
Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
4
votes
4answers
669 views
Text between two tags
I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc.
Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...
3
votes
2answers
367 views
In `while IFS= read..`, why does IFS have no effect?
I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect.
The outer IFS (outside the while ...
2
votes
4answers
184 views
Shell script or command to Strip text and Binary files excerpts from large Log Files
I need to remove a large binary file (PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the ...
16
votes
7answers
20k views
Looping through files with spaces in the names?
I wrote the following script to diff the outputs of two directores with all the same files in them as such:
#!/bin/bash
for file in `find . -name "*.csv"`
do
echo "file = $file";
diff ...
24
votes
8answers
3k views
how do you sort du output by size?
How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong.
[~]# du -sh /var/* | sort -n
0 /var/mail
1.2M ...
7
votes
5answers
2k views
Remove comma between the quotes only in a comma delimited file
I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row
123,"ABC, DEV 23",345,534.202,NAME
I need to ...
6
votes
4answers
3k views
How do I recursively grep through compressed archives?
I'm trying to find out what modules use Test::Version in cpan. So I've used minicpan to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files ...
2
votes
2answers
1k views
Show only text between 2 matching pattern
I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed
If param1 is line matching pattern from other file using sed, below command will works ...
1
vote
3answers
268 views
How to bulk convert all the file in a file system branch between Unix and Windows line break format?
Everybody knows :-) that in Windows plain text files lines are terminated with CR+LF, and in Unix&Linux - with LF only. How can I quickly convert all my source code files from one format to ...
78
votes
5answers
20k views
Why is printf better than echo?
I have heard that printf is better than echo and I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL ...
12
votes
3answers
4k views
Can grep output only specified groupings that match?
Say I have a file:
# file: 'test.txt'
foobar bash 1
bash
foobar happy
foobar
I only want to know what words appear after "foobar", so I can use this regex:
"foobar \(\w\+\)"
The parenthesis ...
3
votes
2answers
533 views
How to do a text replacement in a big folder hierarchy?
I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not.
Something similar to vim's ...
11
votes
6answers
10k views
How to cut part from log file?
I have 8 Gb long log file (Rails production log), and I need to cut part between some dates (lines). Which command i have to use, to do this?
7
votes
6answers
589 views
Combining 2 different cut outputs in a single command?
I have a file such as the following:
1234
ABCD
EFGH
I'd like to convert it to the following:
2341
BCDA
FGHE
The actual file has 4,000 words, so I would like to do this in an efficient manner. I ...
10
votes
2answers
897 views
How to get only the unique results without having to sort data?
$ cat data.txt
aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb
$ cat data.txt | uniq
aaaaaa
cccccc
aaaaaa
bbbbbb
$ cat data.txt | sort | uniq
aaaaaa
bbbbbb
cccccc
$
The result that I need is to display ...
5
votes
4answers
568 views
A standard tool to convert a byte-count into human KiB MiB etc; like du, ls1
Is there a standard tool which converts an integer count of Bytes into a human-readable count of the largest possible unit-size, while keeping the numeric value between 1.00 and 1023.99 ?
I have my ...
3
votes
3answers
836 views
Fast way to build a test file with every second listed in YYYY-mm-dd HH:MM:SS format
I want to create a large test file with lines containg dates listed by the second, but my method is taking inordinately long... (or at least, that's how it feels :) ... 43 minutes to create only ...
11
votes
6answers
1k views
Command to display first few and last few lines of a file
I have a file with many rows, and each row has a timestamp at the starting, like
[Thread-3] (21/09/12 06:17:38:672) logged message from code.....
So, I frequently check 2 things from this log file.
...
7
votes
2answers
322 views
wc -L reports a line-length of 8 for a tab-char. bug or feature?
-L is a useful feature of wc, or so I thought. It prints the length of the longest line. For some reason it expands a single-byte tab-char to a length of 8.
Is there some way to set this to not ...
6
votes
5answers
981 views
Why does shell Command Substitution gobble up a trailing newline char?
As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens
x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p
# ...
5
votes
3answers
12k views
How to use regrex with AWK for string replacement in this example?
Suppose there is some text from a file:
(bookmarks
("Chapter 1 Introduction 1" "#1"
("1.1 Problem Statement and Basic Definitions 23" "#2")
("Exercises 31" "#30")
("Notes and References 42" "#34"))
)
...
4
votes
3answers
248 views
Delete the matching line and several more from a file
I have a text file called file_a.txt.
My first command is
grep -A 12 ".production =" file_a.txt
The output is a few block.
Each block of string contains 13 rows
I specifically want to delete all ...
3
votes
3answers
613 views
Inserting text at the beginning of a file with sed via the terminal in Linux [duplicate]
Possible Duplicate:
How can I prepend a tag to the beginning of several files?
How do I insert text at the beginning of a file via terminal?
1
vote
4answers
335 views
Replace text between brackets
I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like:
SOMETHING [BAR, WIBBLE]
SOMETHING [BAR]
This gives the desired result of:
SOMETHING [FOO, FOO]
SOMETHING [FOO]
...
13
votes
5answers
3k views
sort but keep header line in the at the top?
I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
11
votes
2answers
654 views
echo a file without the first and last lines
Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
8
votes
4answers
2k views
Efficient in-place header removing for large files using sed?
The commands below may takes minutes depends on the file size. Is there any more effient method?
sed -i 1d large_file
7
votes
6answers
1k views
Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?
grep -A 2 -B 3
prints 2 lines after the grep string and prints 3 lines before.
grep -C 3
prints 3 Lines before and 3 lines after
Unfortunately, the grep I'm using does not support these ...
12
votes
2answers
5k views
How do I remove certain lines (using line numbers) in a file?
There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
12
votes
9answers
1k views
There must be a better way to replace single newlines only?
I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
10
votes
2answers
484 views
Where has the trailing newline char gone from my command substitution?
The following code best describes the situation. Why is the last line not outputting the trailing newline char? Each line's output is shown in the comment. I'm using GNU bash, version 4.1.5
...
9
votes
3answers
355 views
Process last line first using awk
I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally.
The ...
6
votes
1answer
246 views
Is there a way to make perl -i not clobber symlinks?
A friend of mine points out that if you do:
perl -pi.bak -e 's/foo/bar/' somefile
when "somefile" is actually a symlink, perl does just what the docs say it will do:
It does this by renaming ...
4
votes
2answers
2k views
How do I insert a space every four characters in a long line?
I've got a long line that I want to insert a space every 4 characters, on a single lone line of solid text to make it easier to read, what's the simplest way to do this? also I should be able to input ...
3
votes
5answers
180 views
evaluate multiple patterns from program output and write into pattern specific files
I have a script outputting some value/numbers and I want to split those into two files. I am looking at something like:
./runme.sh | grep 'ook' >> ook.out | grep 'eek' >> eek.out
Where ...
9
votes
3answers
404 views
Doing two things with output from a command
I have a program texcount that outputs the number of words in my LaTeX document. I can also pipe the output of this to sed to make the newlines TeX linebreaks and write this to a file which I can then ...
8
votes
5answers
389 views
What's the best way to take a segment out of a text file?
What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
7
votes
4answers
617 views
How do I find the overlap of two strings in bash?
I have two strings. For the sake of the example they are set like this:
string1="test toast"
string2="test test"
What I want is to find the overlap starting at the beginning of the strings. With ...
6
votes
4answers
969 views
How to prepend a license header recursively for all .h and .cpp files in a directory
I'm trying to add a license header to all header files and source files in a project directory using a for loop. This is not working, is there any other approach using sed?
6
votes
4answers
1k views
`highlight` command
Is there a command which can be used to easily highlight (with colors, bold text or inverted back-/foreground) specific strings in stdin while passing the entire string to stdout? I'm thinking of ...
5
votes
2answers
210 views
How to cut (select) a field from text line counting from the end?
I know how to select a field from a line using the cut command. For instance, given the following data:
a,b,c,d,e
f,g,h,i,j
k,l,m,n,o
This command:
cut -d, -f2 # returns the second field of the ...
4
votes
4answers
439 views
A unix command to truncate each line of a file
I have a CSV file and I want to truncate it from the third semicolon. For example, if I have this file:
1;foo;bar;baz;x;y;z
2;foo;bar;baz;x;y;z
3;foo;bar;baz;x;y;z
I want to get the following ...
4
votes
1answer
499 views
Is there a search utility which can binary split search a large sorted (sequential) text file?
Yes, I know it sounds odd. Sequential and Binary-splits don't mix.. That is unless the sequence is the byte offset within the file itself...
I've scrambled together a binary split search in bash ...
