Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

5
votes
4answers
210 views

reverse order of paragraphs in file

I have a file containing text in paragraphs (lines with text separated by one or more empty lines). I would like to reverse the order of paragraphs (i.e. last paragraph will become the first, ...), ...
5
votes
5answers
958 views

How to append data of multiple files into single file?

For example: I have 01.txt and 02.txt files and need to append data of these files into new file new.txt. It should append data based on 01.txt followed by 02.txt files. Before this I have to delete ...
5
votes
1answer
386 views

inexact text search

Is there any utility like grep or even uniq but for inexact search or I should write it myself? I mean it will look at 90% (number may vary) matching, or smth like that. For example I have file with ...
5
votes
5answers
15k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
5
votes
3answers
1k views

Fast way to build a test file with every second listed in YYYY-mm-dd HH:MM:SS format

I want to create a large test file with lines containg dates listed by the second, but my method is taking inordinately long... (or at least, that's how it feels :) ... 43 minutes to create only ...
5
votes
2answers
1k views

sort output by column

I'd like to take this command find -maxdepth 1 -type d | while read -r dir; do printf "%s:\t" "$dir"; find "$dir" | wc -l; done ( from here ). which has an output of basically ./kennel: 11062 ...
5
votes
4answers
221 views

How could I simplify this command to only use awk?

awk '/user/ {print $1 }' /etc/userdomains | sed 's/://' the format of /etc/userdomains is domain.tld: user otherdomain.tld: otheruser
5
votes
4answers
4k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
5
votes
3answers
204 views

removing redundancy from output columns

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns A B B A A C A D C A D A B C C B A B and B A ...
5
votes
2answers
388 views

Find all files with a Python Shebang

I'm trying to run a PEP8 check against a large source tree. The tree is comprised of a mix of files in various languages. The idea is to check all the Python scripts without having to explicitly list ...
5
votes
2answers
1k views

How to delete commands in history matching a given string?

I need to delete all commands in my history matching a string. I've tried: $ history | grep searchstring | cut -d" " -f2 | history -d -bash: history: -d: option requires an argument $ history | grep ...
5
votes
3answers
2k views

How can sed output be formatted like printf's formatted printing?

Can sed replace text with a string formatted like printf's formatted printing? The following sed command replaces a line starting with the current value of "$domain" with several values specified in ...
5
votes
3answers
5k views

How to insert a line into text document right before line containing some text in bash?

I have a variable say $strToInsert I have a file file.html I wonder how to find last appearence of </head> and insert a new line before line with it and fill it with $strToInsert contents? ...
5
votes
4answers
1k views

Any (text based) plotting tools in unix toolbox?

I was wondering if there are any generic plotting tools available in the unix toolbox? By plotting I mean simple bars or things you see in many curses based utilities. And by unix toolbox I mean ...
5
votes
2answers
255 views

bash: colorize second column of output

let's say, I have following output from ls: $ ls -lAhF /bin -rwxr-xr-x 1 root root 905K Apr 10 2010 bash* -rwxr-xr-x 3 root root 31K Dec 26 2011 bunzip2* -rwxr-xr-x 1 root root 505K Nov 15 2010 ...
5
votes
2answers
689 views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
5
votes
4answers
5k views

Text between two tags

I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...
5
votes
4answers
913 views

Tool to create text files from a template

I have to regularly create 100+ text files from templates. I currently use an overly complicated shell script. I think there is a smarter way to handle this, but I don't know how. I have a "database" ...
5
votes
2answers
1k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
5
votes
4answers
210 views

How to grep blocks? Or I should use awk/ack?

Say I have a file with information as follows: ... Entry '234238': some text some text some text some text Entry '899823': some text some text some text Entry '234238': more text more ...
5
votes
3answers
285 views

Extracting fixed width records with no delimiter from a single line

I need to extract strings of text from a single file containing one very long line of text with no delimiters. Using the sample line below, these are the following known facts: ...
5
votes
1answer
207 views

How to remove all instances of a particular symbol from a text file?

I've got a huge (about half a GiB, impossible to use a usual text editor on) CSV file with fields enclosed in double quotes like "abc","def" but need a file without quotes (I am sure this is not going ...
5
votes
3answers
557 views

Extracting lines by key from very large file

I have a 42M line text file. Th first nine characters of each line are a numeric key. What is the most efficient way to extract only the lines whose key exists in another list of about 1.5M keys? Both ...
5
votes
6answers
1k views

How to sort (by whatever key) BibTex entries in `.bib` file?

I work with LaTeX and do versioning with Git. For bibliography management I use Mendeley. The problem is that each time Mendeley synchronizes it's .bib exports, they are in different order, what ...
5
votes
3answers
2k views

Command like `column -t` that instead keeps separators in output

I'm editing a simple table. I would like to have it nicely formatted. While I could use tbl, latex, or similar, this seems overkill -- plain text really is sufficient. As it's simple I might as ...
5
votes
2answers
2k views

Count the number of words of particular length from a file

I've used the below command to count the number of words in a file: tr ' ' '\n' < Filename | grep -c "WORD" This returns the word list with counter. Now I want to count the number of words of a ...
5
votes
2answers
921 views

Single command taking two strings to extract string between them, like 'tr' (without expressions)

Saw here a way to use sed to get text between two other strings in a line, like: sed 's/.*starting_text\(.*\)ending_text.*/\1/' but I'd like a simple command (like tr, but for string extraction) ...
5
votes
4answers
1k views

SED : Delete 4 line above and below 5 line after pattern match

I have Hugh file with following details : define host{ use generic-host ; Name of host template to use host_name ServerA_172.29.16.102 ...
5
votes
2answers
173 views

awk + paste for cleaning up PATH?

I have seen this code in .cshrc init files on a few machines. I went through a few awk tutorials in trying to understand how it works, but I am still unable to decrypt it. setenv PATH `echo $PATH | ...
5
votes
2answers
3k views

How does one find and replace text in all open files with kate?

How does one find and replace text in all open files with kate?
5
votes
2answers
258 views

Convert between Unicode Normalization Forms on the unix command-line

In Unicode, some character combinations have more than one representation. For example, the character ä can be represented as "ä", that is the codepoint U+00E4 (two bytes c3 a4 in UTF-8 encoding), ...
5
votes
2answers
130 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
5
votes
1answer
109 views

Inderect references aka value of a value in awk

I'm searching for a way to solve this problem in awk. Input: X 1 Y 2 Z 3 X 4 Y 5 The output should look like this: X 5 Y 7 Z 3 I see two problems here: the first one is the indirect reference in ...
5
votes
4answers
778 views

Compare an old file and new file, but ignore lines which only exist in new file?

I have two files: oldlist - This contains a list of files and a md5 hash for each file. This was generated one year ago. newlist - This also contains a list of files and a md5 hash for each file. ...
5
votes
1answer
117 views

array[@] output all messed up?

I've got this code: Unix+=("Stock List") while read line; do result=$(wget -O - -o /dev/null "http://finance.yahoo.com/d/quotes.csv?s=$line&f=sl1&e=.csv" | tr ',' ' ' | tr '"' ' ') ...
5
votes
1answer
104 views

Is there a command like head but for line truncation?

Linux's head selects a number of lines from the top of a file. Is there a command that selects a number of characters from the front of a line? The use case behind the question is this: I want to ...
5
votes
1answer
111 views

Search for a text in Linux but don't display the complete line in results

I'm using grep to do this, like: grep someText file.log But this returns the complete line in results. If someText is found in the middle of a line I don't want to ouput the whole line, just the ...
5
votes
3answers
269 views

Searching a text file with a single line using regular expressions

as far as i know all unix text processing utilities are reading one line at a time and performing one action on this line. I have a huge file with a single line of text which contains several tokens ...
5
votes
5answers
576 views

Parsing a delimited text file in bash as command arguments

I have a text file split up like so: field1,field2,field3 xield1,xield2,xield3 dield1,dield2,dield3 gield1,gield2,gield3 Each of these columns will be a parameter to a program, and I would like ...
5
votes
2answers
181 views

Lowercase all but the first (Uppercase) letter from UPPERCASE in cyrillic

To make all letters a lower case except the first letter. The first letter would look like "Uppercase" after I changed (from UPPERCASE in cyrillic). The rest (not UPPERCASED) leave unchanged. I'm ...
5
votes
3answers
217 views

What's the easiest way to make a list of most common words in a list?

Say I have a bunch of textfiles containing fiction, non-fiction, newspaper articles, &c (random examples of text in a given language.) I want a frequency list of the given words, most common word ...
5
votes
2answers
336 views

sort every field numerically, varying field counts

I'm trying to sort some data using sort. I noticed it was sorting by digit rather than number, so I added the -n flag. It then seemingly only numerically sorts on the first field though. Breaking it ...
5
votes
2answers
44 views

Sort one column only, keeping remaining columns untouched

How to (numerically) sort a specific column in a text file, without affecting other columns (irrespective of whether they are sorted or unsorted)? On other threads I found -s argument, but it does ...
5
votes
1answer
97 views

counting multiple patterns in a single pass with grep?

I've written a grep loop to iteratively count DNA trinucleotides within a gzipped DNA fasta file containing DNA sequences e.g. declare -a tri=(AAA AAC AAG AAT CAA .. etc) for i in ${tri[@]} do ...
5
votes
5answers
258 views

Split the content of a file in linux

I have a text file which has content like this: abc.tar^@xxx.tar^@yyy.tar^@ Say for example I have this content in a file named abc.txt and I want to split the content and write the first two ...
5
votes
3answers
121 views

Filtering redundant paths from a list of paths

I have a list of relative paths such as this: dir1 dir2 dir2/dir3 dir2/file1 dir3/file2 dir3/dir4 dir3/dir4/file3 In the example above, the specifier dir2/file1 (for example) is redundant, because ...
5
votes
1answer
277 views

merge that works at word granularity instead of line

The old diff, patch, and merge programs, and their modern descendents in the form of version control systems work great for managing source code that can be edited concurrently by multiple people. ...
5
votes
1answer
36 views

How to display the non-sparse parts of a sparse file?

Imagine a file created with: truncate -s1T file echo test >> file truncate -s2T file I now have a 2 tebibyte file (that occupies 4kiB on disk), with "test\n" written in the middle. How would ...
5
votes
1answer
99 views

How to append multiple lines to a file without last newline?

I have a very long string that is split in chunks. I want to append them to a file without putting the newline character using bash. Example: First append cat >> abc.sh << EOL echo "bla ...
5
votes
4answers
42 views

Increment numbers inside strings by an offset

I want to search a text file for strings like these. [task0] [task1] ... [task1613] and increment each by a given offset, in this case by 3640. So, with the example I gave above, I'd get the ...