Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

7
votes
4answers
254 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
7
votes
2answers
1k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
7
votes
2answers
928 views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
7
votes
4answers
327 views

Getting text from last marker to EOF in POSIX.2

I have a text with marker lines like: aaa --- bbb --- ccc I need to get a text from the last marker (not inclusive) to EOF. In this case it will be ccc Is there an elegant way within POSIX.2? ...
7
votes
5answers
5k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
7
votes
4answers
251 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...
7
votes
2answers
2k views

How can I use column to delimit on tabs and not spaces?

I'd like to use Unix column to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I make column ...
7
votes
5answers
605 views

How intelligent are unix utilities?

Unix utilities like sort, find, grep, diff et al are very handy to perform quick tasks, sometimes without writing any code at all. I wanted to know what algorithms do they internally use and how do ...
7
votes
1answer
3k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
7
votes
2answers
712 views

What governs the limits of shell brace expansion?

In this example I refer to the expansion of a sequence of integers, but perhaps(?) the limits would be relevant to all aspects of brace expansion.. This more general view is also of interest to me. ...
7
votes
2answers
7k views

How to find difference between two timestamps up to milliseconds?

I am new to shell scripting. The heart of my script is to find the difference between two timestamps up to milliseconds. With me I have a file with content of timestamps only as 2012-09-13 ...
7
votes
1answer
320 views

Change date “July 29th, 2011” to “20110729”

I have lots of HTML files contains date with format July 29th, 2011 I want to change date format July 29th, 2011 to 20110729, December 9th, 2010 to 20101209, etc. I think sed may works but still ...
7
votes
1answer
71 views

How to display the non-sparse parts of a sparse file?

Imagine a file created with: truncate -s1T file echo test >> file truncate -s2T file I now have a 2 tebibyte file (that occupies 4kiB on disk), with "test\n" written in the middle. How would ...
7
votes
4answers
878 views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
7
votes
3answers
2k views

Is it possible to nest a 'find -exec' within another 'find -exec'?

Something like the following is what I what I'm after, but my code doesn't work, no matter how I escape {} and + ; find ./ -maxdepth 1 -type d -name '.*' -exec \ find {} -maxdepth 1 -type f ...
7
votes
2answers
145 views

How does one find and replace text in all open files with jed?

How does one find and replace text in all open files with jed?
7
votes
2answers
416 views

Strip // Comments From Files

What's the best way to strip all code comments in a given directory? I'd like to strip out all // ... EOL comments, and /* blah \*/ (or /** ... \*/) comments as well. This is a PHP project, and I'd ...
7
votes
0answers
113 views

How to filter any part of description or option from manpage [closed]

I want to get output of any desired description or option from a man page. Example: I want to get the description for the autoclean option from the apt-get man page: Please suggest a command, ...
6
votes
7answers
657 views

Removing a specific line from a file

Somewhere in the middle of my CSV file is this line: Products below this line are out of stockNumber, month, year, reference, store Note:Number, month, year, reference and store are the CSV fields. ...
6
votes
8answers
742 views

How do I construct a list with numbers like this?

How do I construct a list like this: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Where I run command 15 or something. Or if I specify 100 it would make it with 100 numbers or 10000 and it would make it ...
6
votes
5answers
622 views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to other file, without sed and awk? For example, $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i ...
6
votes
10answers
2k views

Copy only Specific text of a file to another

I have a file abc.txt the contents are <classpathentry kind="src" path="Sources"/> <classpathentry kind="con" path="WOFramework/ERExtensions"/> <classpathentry kind="con" ...
6
votes
5answers
262 views

Get filesystem name and usage with `df -h`

df -h | grep test | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: /test /test/drv0 /test/drv1 df -h | grep test | cut -d '%' -f1 | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: 10 20 15 I want to ...
6
votes
7answers
17k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
6
votes
8answers
2k views

Grep rest of line…after match

I have a file containing only two lines, with the following structure: $ cat /tmp/pwpower.log 000D6F0000D34227, -114.10 000D6F0001A405C4, -130.09 The values are power values of my solar plant. ...
6
votes
4answers
1k views

uniq a csv file ignoring a column, awk maybe?

Given this file (annotations are not part of file, but form part of explanation)... x,a,001,b,c,d,y x,a,002,b,c,e,yy x,bb,003,b,d,e,y x,c,004,b,d,e,y x,c,005,b,d,e,y # nb - dupe of row 4 ...
6
votes
9answers
3k views

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through unix.stackexchange.com and on the ...
6
votes
3answers
6k views

How do I insert a space every four characters in a long line?

I've got a long line that I want to insert a space every 4 characters, on a single lone line of solid text to make it easier to read, what's the simplest way to do this? also I should be able to input ...
6
votes
7answers
9k views

How to test whether the file is crlf or lf without modyfing it?

I need to periodically run a command, that ensures that some text files are kept in linux mode. Unfortunately dos2unix always modifies file, which would mess file's and folder's timestamp and cause ...
6
votes
7answers
580 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
6
votes
3answers
840 views

Counting the occurrences of the string

As input, we have some ids which are strings. How do we calculate what identifier has appeared most frequently and how many times? Example Input: cat dog cat dog dog dog dog fly spider spider ...
6
votes
4answers
273 views

How to remove duplicate characters?

If I am having a line as: Thhiisss iisss mmyyy nameeee I want to print this as: This is my name What is the unix command for this?
6
votes
3answers
15k views

remove particular characters from a variable using bash

I want to parse a variable (in my case it's development kit version) to make it dot(.) free. If version='2.3.3', desired output is 233. I tried as below, but it requires . to be replaced with another ...
6
votes
4answers
18k views

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5
6
votes
3answers
310 views

Shell script to search files for identical text entries

Here's a nice teaser for a shell scripting guru: Take a directory with several text files. Can be a few up to ~1000. All files contain an identifier on a given line (always the same line). Identify ...
6
votes
7answers
695 views

Count total number of lines before/after a pattern match

I am having a long list of IP addresses, which are not in sequence. I need to find how many IP addresses are there before/after a particular IP address. How can I achieve this?
6
votes
4answers
654 views

We need a “C” list that contains all the elements of “A”, but doesn't have any of “B”

We have two lists. A bigger "A": A=`echo -e '1\n2\n3\n4\n5'` echo "$A" 1 2 3 4 5 and a smaller "B": B=`echo -e '1\n2\n3'` echo "$B" 1 2 3 Q: But we need a third list that contains all the ...
6
votes
4answers
19k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
6
votes
3answers
215 views

sed with multiple expression for in-place arguement

I am trying to replace multiple words in file by using sed -i #expression1 #expression2 file Something 123 item1 Something 456 item2 Something 768 item3 Something 353 item4 Output (Desired) ...
6
votes
4answers
293 views

How can I print a conditional header BEFORE stdout, if there is any output on stdout

I have a process that filters a list of files from a directory (having find check to see if there are files older than a certain period to show a queue is stuck). It may or may not return anything, ...
6
votes
3answers
30k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
6
votes
2answers
4k views

grep surrounding characters of a match

I'm looking to do a find and replace within a giant database dump, and it's not doing what I think should happen. I'd like to grep for my target string in the file, and then see the surrounding 8 ...
6
votes
5answers
21k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
6
votes
4answers
183 views

Need to move the last line of the file to second line of the same file

I have a file 'test' in linux with below data. aaaaaa bbbbbb cccccc dddddd eeeeee I need to cut the last line of this file and put it in the second position. It must look like below. aaaaaa eeeeee ...
6
votes
3answers
604 views

How to have command history with timestamps output to the terminal continuously?

I use a simple alias to enable the "tracking" of commands in one or many terminal window(s): alias trackmi='export PROMPT_COMMAND="history -a; $PROMPT_COMMAND"' Then I just tail -f my .bash_history ...
6
votes
5answers
7k views

Transposing rows and columns

I have a file with the lines like below. title1:A1 title2:A2 title3:A3 title4:A4 title5:A5 title1:B1 title2:B2 title3:B3 title4:B4 title5:B5 title1:C1 title2:C2 title3:C3 title4:C4 title5:C5 ...
6
votes
4answers
6k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
6
votes
2answers
485 views

Compare two files and print matches

Is there anyway to get this through Unix shell scripting? I have a fileA with one column (1000 rows), and fileB with 26 columns(13000 rows). I need to search each value of fileA with fileB and return ...
6
votes
1answer
551 views

inexact text search

Is there any utility like grep or even uniq but for inexact search or I should write it myself? I mean it will look at 90% (number may vary) matching, or smth like that. For example I have file with ...
6
votes
2answers
848 views

Find all files with a Python Shebang

I'm trying to run a PEP8 check against a large source tree. The tree is comprised of a mix of files in various languages. The idea is to check all the Python scripts without having to explicitly list ...