Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

7
votes
8answers
635 views

Using 'grep' to find lines that contain all of three specified characters in any order [duplicate]

We have a document which contains lines and we have to find if [a|b|c] appears at least once in every line no matter which order. For example: Input: abc bca cab hhfdhdhfabjfdjdjff acjfdjdfjdf ...
7
votes
7answers
812 views

Slick one-liner to convert a list like “1: 2, 3, 4, 5” to “1.2, 1.3, 1.4, 1.5”

Let's say I have a file that looks something like this: 23: a, b, c, d 24: b, d, f 25: c, g and I want to get output like this: 23.a 23.b 23.c 23.d 24.b 24.d 24.f 25.c 25.g Of course it's not ...
7
votes
10answers
4k views

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through unix.stackexchange.com and on the ...
7
votes
10answers
2k views

What's a good way to filter a text file to remove empty lines?

I have a .csv file (on a mac) that has a bunch of empty lines, e.g.: "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
7
votes
2answers
2k views

How can I cut a large file in place? [duplicate]

I have a very big text file, about 80GB, and I need to cut a part form it that lies between two given lines. The part I need is not big, and I have not enough space left on the hard drive to do things ...
7
votes
5answers
9k views

Transposing rows and columns

I have a file with the lines like below. title1:A1 title2:A2 title3:A3 title4:A4 title5:A5 title1:B1 title2:B2 title3:B3 title4:B4 title5:B5 title1:C1 title2:C2 title3:C3 title4:C4 title5:C5 ...
7
votes
5answers
454 views

List out strings which are substrings of other strings in the list

I have a list of names like so: dog_bone dog_collar dragon cool_dragon lion lion_trainer dog I need to extract out names that appear in other names like so: dragon lion dog I looked through the ...
7
votes
6answers
512 views

Obtain values of a section of a file

I have a configuration file in the following format. <Title> [part1] A.File = a A.Val = val1 B.File = a B.Val = val1 [part2] A.File = a1 A.Val = val2 B.File = a B.Val = val1 ...
7
votes
2answers
3k views

A shell script for joining two files

I want to write a shell script that get two files A and B, and get a result like this: File A: user_a tel_a addr_a user_b tel_b addr_b File B: process_1 user_a process_2 user_a process_3 user_b ...
7
votes
4answers
289 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
7
votes
4answers
6k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
7
votes
2answers
1k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
7
votes
2answers
992 views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
7
votes
4answers
332 views

Getting text from last marker to EOF in POSIX.2

I have a text with marker lines like: aaa --- bbb --- ccc I need to get a text from the last marker (not inclusive) to EOF. In this case it will be ccc Is there an elegant way within POSIX.2? ...
7
votes
5answers
6k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
7
votes
4answers
271 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...
7
votes
4answers
16k views

Text between two tags

I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...
7
votes
5answers
605 views

How intelligent are unix utilities?

Unix utilities like sort, find, grep, diff et al are very handy to perform quick tasks, sometimes without writing any code at all. I wanted to know what algorithms do they internally use and how do ...
7
votes
2answers
7k views

How to find difference between two timestamps up to milliseconds?

I am new to shell scripting. The heart of my script is to find the difference between two timestamps up to milliseconds. With me I have a file with content of timestamps only as 2012-09-13 ...
7
votes
2answers
743 views

What governs the limits of shell brace expansion?

In this example I refer to the expansion of a sequence of integers, but perhaps(?) the limits would be relevant to all aspects of brace expansion.. This more general view is also of interest to me. ...
7
votes
1answer
323 views

Change date “July 29th, 2011” to “20110729”

I have lots of HTML files contains date with format July 29th, 2011 I want to change date format July 29th, 2011 to 20110729, December 9th, 2010 to 20101209, etc. I think sed may works but still ...
7
votes
1answer
73 views

How to display the non-sparse parts of a sparse file?

Imagine a file created with: truncate -s1T file echo test >> file truncate -s2T file I now have a 2 tebibyte file (that occupies 4kiB on disk), with "test\n" written in the middle. How would ...
7
votes
4answers
940 views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
7
votes
2answers
146 views

How does one find and replace text in all open files with jed?

How does one find and replace text in all open files with jed?
7
votes
2answers
435 views

Strip // Comments From Files

What's the best way to strip all code comments in a given directory? I'd like to strip out all // ... EOL comments, and /* blah \*/ (or /** ... \*/) comments as well. This is a PHP project, and I'd ...
7
votes
0answers
113 views

How to filter any part of description or option from manpage [closed]

I want to get output of any desired description or option from a man page. Example: I want to get the description for the autoclean option from the apt-get man page: Please suggest a command, ...
6
votes
7answers
678 views

Removing a specific line from a file

Somewhere in the middle of my CSV file is this line: Products below this line are out of stockNumber, month, year, reference, store Note:Number, month, year, reference and store are the CSV fields. ...
6
votes
8answers
746 views

How do I construct a list with numbers like this?

How do I construct a list like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Where I run command 15 or something. Or if I specify 100 it would make it with 100 numbers or 10000 and it would make it ...
6
votes
5answers
653 views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to other file, without sed and awk? For example, $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i ...
6
votes
10answers
3k views

Copy only Specific text of a file to another

I have a file abc.txt the contents are <classpathentry kind="src" path="Sources"/> <classpathentry kind="con" path="WOFramework/ERExtensions"/> <classpathentry kind="con" ...
6
votes
7answers
475 views

How can I double the newlines in an output stream

When I have an output stream: a b c d e How can I double the newlines: a b c d e
6
votes
5answers
269 views

Get filesystem name and usage with `df -h`

df -h | grep test | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: /test /test/drv0 /test/drv1 df -h | grep test | cut -d '%' -f1 | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: 10 20 15 I want to ...
6
votes
7answers
5k views

How to find a position of a character using grep?

I need to identify the postion of a character in string using grep command. Example, the string is RAMSITALSKHMAN|1223333. grep -n '[^a-zA-Z0-9\$\~\%\#\^]' How do I find the position of | in the ...
6
votes
7answers
19k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
6
votes
8answers
2k views

Grep rest of line…after match

I have a file containing only two lines, with the following structure: $ cat /tmp/pwpower.log 000D6F0000D34227, -114.10 000D6F0001A405C4, -130.09 The values are power values of my solar plant. ...
6
votes
4answers
1k views

uniq a csv file ignoring a column, awk maybe?

Given this file (annotations are not part of file, but form part of explanation)... x,a,001,b,c,d,y x,a,002,b,c,e,yy x,bb,003,b,d,e,y x,c,004,b,d,e,y x,c,005,b,d,e,y # nb - dupe of row 4 ...
6
votes
7answers
10k views

How to test whether the file is crlf or lf without modyfing it?

I need to periodically run a command, that ensures that some text files are kept in linux mode. Unfortunately dos2unix always modifies file, which would mess file's and folder's timestamp and cause ...
6
votes
7answers
633 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
6
votes
3answers
1k views

Counting the occurrences of the string

As input, we have some ids which are strings. How do we calculate what identifier has appeared most frequently and how many times? Example Input: cat dog cat dog dog dog dog fly spider spider ...
6
votes
4answers
22k views

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5
6
votes
6answers
18k views

bash - replace space with new line

How can I replace spaces with new lines on an input like: /path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5 etc... To obtain the following: /path/to/file /path/to/file2 ...
6
votes
3answers
328 views

Shell script to search files for identical text entries

Here's a nice teaser for a shell scripting guru: Take a directory with several text files. Can be a few up to ~1000. All files contain an identifier on a given line (always the same line). Identify ...
6
votes
4answers
1k views

Replace a string containing newline characters

With the bash shell, in a file with rows like the following ones first "line" <second>line and so on I would like to replace one or more occurrences of "line"\n<second> with other ...
6
votes
7answers
960 views

Count total number of lines before/after a pattern match

I am having a long list of IP addresses, which are not in sequence. I need to find how many IP addresses are there before/after a particular IP address. How can I achieve this?
6
votes
4answers
654 views

We need a “C” list that contains all the elements of “A”, but doesn't have any of “B”

We have two lists. A bigger "A": A=`echo -e '1\n2\n3\n4\n5'` echo "$A" 1 2 3 4 5 and a smaller "B": B=`echo -e '1\n2\n3'` echo "$B" 1 2 3 Q: But we need a third list that contains all the ...
6
votes
3answers
227 views

sed with multiple expression for in-place arguement

I am trying to replace multiple words in file by using sed -i #expression1 #expression2 file Something 123 item1 Something 456 item2 Something 768 item3 Something 353 item4 Output (Desired) ...
6
votes
4answers
234 views

Need to move the last line of the file to second line of the same file

I have a file 'test' in linux with below data. aaaaaa bbbbbb cccccc dddddd eeeeee I need to cut the last line of this file and put it in the second position. It must look like below. aaaaaa eeeeee ...
6
votes
4answers
301 views

How can I print a conditional header BEFORE stdout, if there is any output on stdout

I have a process that filters a list of files from a directory (having find check to see if there are files older than a certain period to show a queue is stuck). It may or may not return anything, ...
6
votes
3answers
32k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
6
votes
2answers
4k views

grep surrounding characters of a match

I'm looking to do a find and replace within a giant database dump, and it's not doing what I think should happen. I'd like to grep for my target string in the file, and then see the surrounding 8 ...