Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

7
votes
5answers
5k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
7
votes
4answers
228 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...
7
votes
3answers
3k views

How do I make python programs behave like proper unix tools?

I have a few Python scripts laying around, and I'm working on rewriting them. I have the same problem with all of them. It's not obvious to me how to write the programs so that they behave like ...
7
votes
5answers
605 views

How intelligent are unix utilities?

Unix utilities like sort, find, grep, diff et al are very handy to perform quick tasks, sometimes without writing any code at all. I wanted to know what algorithms do they internally use and how do ...
7
votes
1answer
3k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
7
votes
2answers
689 views

What governs the limits of shell brace expansion?

In this example I refer to the expansion of a sequence of integers, but perhaps(?) the limits would be relevant to all aspects of brace expansion.. This more general view is also of interest to me. ...
7
votes
2answers
7k views

How to find difference between two timestamps up to milliseconds?

I am new to shell scripting. The heart of my script is to find the difference between two timestamps up to milliseconds. With me I have a file with content of timestamps only as 2012-09-13 ...
7
votes
1answer
319 views

Change date “July 29th, 2011” to “20110729”

I have lots of HTML files contains date with format July 29th, 2011 I want to change date format July 29th, 2011 to 20110729, December 9th, 2010 to 20101209, etc. I think sed may works but still ...
7
votes
1answer
68 views

How to display the non-sparse parts of a sparse file?

Imagine a file created with: truncate -s1T file echo test >> file truncate -s2T file I now have a 2 tebibyte file (that occupies 4kiB on disk), with "test\n" written in the middle. How would ...
7
votes
4answers
825 views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
7
votes
3answers
2k views

Is it possible to nest a 'find -exec' within another 'find -exec'?

Something like the following is what I what I'm after, but my code doesn't work, no matter how I escape {} and + ; find ./ -maxdepth 1 -type d -name '.*' -exec \ find {} -maxdepth 1 -type f ...
7
votes
2answers
143 views

How does one find and replace text in all open files with jed?

How does one find and replace text in all open files with jed?
7
votes
2answers
400 views

Strip // Comments From Files

What's the best way to strip all code comments in a given directory? I'd like to strip out all // ... EOL comments, and /* blah \*/ (or /** ... \*/) comments as well. This is a PHP project, and I'd ...
7
votes
0answers
113 views

How to filter any part of description or option from manpage [on hold]

I want to get output of any desired description or option from a man page. Example: I want to get the description for the autoclean option from the apt-get man page: Please suggest a command, ...
6
votes
7answers
656 views

Removing a specific line from a file

Somewhere in the middle of my CSV file is this line: Products below this line are out of stockNumber, month, year, reference, store Note:Number, month, year, reference and store are the CSV fields. ...
6
votes
8answers
739 views

How do I construct a list with numbers like this?

How do I construct a list like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Where I run command 15 or something. Or if I specify 100 it would make it with 100 numbers or 10000 and it would make it ...
6
votes
10answers
2k views

Copy only Specific text of a file to another

I have a file abc.txt the contents are <classpathentry kind="src" path="Sources"/> <classpathentry kind="con" path="WOFramework/ERExtensions"/> <classpathentry kind="con" ...
6
votes
5answers
261 views

Get filesystem name and usage with `df -h`

df -h | grep test | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: /test /test/drv0 /test/drv1 df -h | grep test | cut -d '%' -f1 | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: 10 20 15 I want to ...
6
votes
7answers
17k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
6
votes
4answers
591 views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to other file, without sed and awk? For example, $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i ...
6
votes
8answers
1k views

Grep rest of line…after match

I have a file containing only two lines, with the following structure: $ cat /tmp/pwpower.log 000D6F0000D34227, -114.10 000D6F0001A405C4, -130.09 The values are power values of my solar plant. ...
6
votes
4answers
1k views

uniq a csv file ignoring a column, awk maybe?

Given this file (annotations are not part of file, but form part of explanation)... x,a,001,b,c,d,y x,a,002,b,c,e,yy x,bb,003,b,d,e,y x,c,004,b,d,e,y x,c,005,b,d,e,y # nb - dupe of row 4 ...
6
votes
9answers
2k views

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through unix.stackexchange.com and on the ...
6
votes
7answers
8k views

How to test whether the file is crlf or lf without modyfing it?

I need to periodically run a command, that ensures that some text files are kept in linux mode. Unfortunately dos2unix always modifies file, which would mess file's and folder's timestamp and cause ...
6
votes
2answers
6k views

How do I insert a space every four characters in a long line?

I've got a long line that I want to insert a space every 4 characters, on a single lone line of solid text to make it easier to read, what's the simplest way to do this? also I should be able to input ...
6
votes
7answers
530 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
6
votes
3answers
283 views

Shell script to search files for identical text entries

Here's a nice teaser for a shell scripting guru: Take a directory with several text files. Can be a few up to ~1000. All files contain an identifier on a given line (always the same line). Identify ...
6
votes
7answers
596 views

Count total number of lines before/after a pattern match

I am having a long list of IP addresses, which are not in sequence. I need to find how many IP addresses are there before/after a particular IP address. How can I achieve this?
6
votes
4answers
652 views

We need a “C” list that contains all the elements of “A”, but doesn't have any of “B”

We have two lists. A bigger "A": A=`echo -e '1\n2\n3\n4\n5'` echo "$A" 1 2 3 4 5 and a smaller "B": B=`echo -e '1\n2\n3'` echo "$B" 1 2 3 Q: But we need a third list that contains all the ...
6
votes
3answers
12k views

remove particular characters from a variable using bash

I want to parse a variable (in my case it's development kit version) to make it dot(.) free. If version='2.3.3', desired output is 233. I tried as below, but it requires . to be replaced with another ...
6
votes
3answers
210 views

sed with multiple expression for in-place arguement

I am trying to replace multiple words in file by using sed -i #expression1 #expression2 file Something 123 item1 Something 456 item2 Something 768 item3 Something 353 item4 Output (Desired) ...
6
votes
3answers
1k views

Remove only the commas present within the double quotes

In a text file, I want to remove , (commas) and also the " (quotes) (only if the double quotes contains numbers seperated by commas). 56,72,"12,34,54",x,y,"foo,a,b,bar" Expected ouput ...
6
votes
4answers
289 views

How can I print a conditional header BEFORE stdout, if there is any output on stdout

I have a process that filters a list of files from a directory (having find check to see if there are files older than a certain period to show a queue is stuck). It may or may not return anything, ...
6
votes
3answers
29k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
6
votes
2answers
3k views

grep surrounding characters of a match

I'm looking to do a find and replace within a giant database dump, and it's not doing what I think should happen. I'd like to grep for my target string in the file, and then see the surrounding 8 ...
6
votes
5answers
20k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
6
votes
4answers
169 views

Need to move the last line of the file to second line of the same file

I have a file 'test' in linux with below data. aaaaaa bbbbbb cccccc dddddd eeeeee I need to cut the last line of this file and put it in the second position. It must look like below. aaaaaa eeeeee ...
6
votes
3answers
584 views

How to have command history with timestamps output to the terminal continuously?

I use a simple alias to enable the "tracking" of commands in one or many terminal window(s): alias trackmi='export PROMPT_COMMAND="history -a; $PROMPT_COMMAND"' Then I just tail -f my .bash_history ...
6
votes
2answers
386 views

Compare two files and print matches

Is there anyway to get this through Unix shell scripting? I have a fileA with one column (1000 rows), and fileB with 26 columns(13000 rows). I need to search each value of fileA with fileB and return ...
6
votes
1answer
527 views

inexact text search

Is there any utility like grep or even uniq but for inexact search or I should write it myself? I mean it will look at 90% (number may vary) matching, or smth like that. For example I have file with ...
6
votes
2answers
785 views

Find all files with a Python Shebang

I'm trying to run a PEP8 check against a large source tree. The tree is comprised of a mix of files in various languages. The idea is to check all the Python scripts without having to explicitly list ...
6
votes
3answers
8k views

How to insert a line into text document right before line containing some text in bash?

I have a variable say $strToInsert I have a file file.html I wonder how to find last appearence of </head> and insert a new line before line with it and fill it with $strToInsert contents? ...
6
votes
1answer
643 views

How to detect and delete lines containing ˆ@

I have a simple problem: In my file, the are lines containing the string ˆ@ˆ@ˆ@ˆ@ˆ@ˆ@. I just want to delete all lines with this string, using for example the sed or grep commands. And I would like ...
6
votes
4answers
355 views

How to split text between separator into multiple files?

I have a text file contaning the following: "random textA" "random random textB" The separator is " How can I split the containt into multiple file as follow using a bash command? File 1 : ...
6
votes
3answers
711 views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
6
votes
3answers
264 views

Empty a file without grep subsequently treating it like a binary file

Currently I have netcat piping output to tee which is writing to output.txt with nc -l -k -p 9100 | tee output.txt I want to monitor this output, so I'm watching it with tail -f | egrep -i 'regex' ...
6
votes
1answer
274 views

How do I print all but the Nth to last line in sed?

I'd like to do the complement / "opposite" of sed 13q;d <file.txt More generally, is it possible to do this sort of complement/inverse/opposite in sed? Or only for regexes? How do I print all ...
6
votes
2answers
443 views

How to report “sed” in-place changes

When using sed to replace strings in-place, is there a way to make it report the changes it does (without relying on a diff of old and new files)? For instance, how can I change the command line ...
6
votes
2answers
514 views

bash: colorize second column of output

let's say, I have following output from ls: $ ls -lAhF /bin -rwxr-xr-x 1 root root 905K Apr 10 2010 bash* -rwxr-xr-x 3 root root 31K Dec 26 2011 bunzip2* -rwxr-xr-x 1 root root 505K Nov 15 2010 ...
6
votes
4answers
13k views

Text between two tags

I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...