Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

1
vote
7answers
286 views

repeat a line, splitting one field

I have a tab delimited file in which in every line I have this: K00001;K00004;K00008 0 0 34 0 0 0 0 0 0 0 0 0 0 0 0 0 36 0 0 52 0 0 0 6 0 I would ...
1
vote
2answers
228 views

Extracting list of patterns which are output of another command

I can extract list of patterns using following command, fgrep -A 1 -f patternlist.txt filename.fasta but, is there a way I can extract without creating another file (patternlist.txt in this case) ...
3
votes
1answer
30 views

Replacing " with pretty quotes on a lot of files

I have a lot of UTF-8 encoded files with words quoted like this "word" and I want to replace the quotes with “word” (the first quotes are upside down and the second is normal) Because the ...
3
votes
2answers
2k views

Remove escape characters using sed

I have input coming from pipe which contains special escape characters. For illustration, the stream looks like this: printf "foo.bar\033[00m" > file cat file How can I remove the trailing .bar\...
0
votes
2answers
97 views

How to get portion of lines from all .txt files in a directory?

I have 5,000 text files of journal article citations. I am trying to extract only the abstract portion. Meaning that I want to keep the same text document and delete all the other text except for ...
6
votes
3answers
2k views

“parentheses not balanced” even though I escaped it?

Im trying to run this simple sed command sed '/VALUES\ \([0-9]/d!' It still gives me an error saying "RE error: parentheses not balanced" (yes the quotes are there). What can I do? I mean I could ...
0
votes
3answers
203 views

Strip the file base name, keep the directory part

I have a string which looks something like this dir/subdir/othersubdir/file.txt But it can also look like dir/file.txt or dir/subdir/file.txt Now, I am making a script where $P is the full ...
4
votes
3answers
549 views

Copy a specific percentage of each file in a directory to a new file

For example, we have N files (file1, file2, file3 ...) We need first 20% of them, the result directory should be like (file1_20, file2_20, file3_20 ...). I was thinking use wc to get the lines of ...
2
votes
1answer
101 views

Specify lines in awk?

Im going to use this awk script to delete duplicate lines from my text file: awk '!x[$0]++' However, I want to specify which lines that they are to be run on. There could be 10000 duplicates, but ...
0
votes
2answers
1k views

Shell script to add or replace block of text?

I'm looking for a shell script (Bash) that adds or replaces copyright text at the beginning of the file (for .c and .h files). It should search and add or replace the copyright text block for all the ...
2
votes
1answer
268 views

sed: append line and number output

In Linux, say one wants to add the following line before line number five (thus line nuber 5 becomes line number 6) and wants to have all the lines in the output numbered throught the sed 'tool' = ...
2
votes
8answers
11k views

How to print lines with blank 5th field in CSV

I have to check if the fifth field is empty in a CSV file. This is my file: 1,abc,543,87,DATA,fsg; 1,abc,543,87,,fsg; 1,abc,543,87,DATA,fsg; 1,abc,543,88,,fsg; 1,abc,543,,DATA,fsg; As you can ...
21
votes
5answers
26k views

Convert file contents to lower case

I have temp file with some lower-case and upper-case contents. Input Contents of my temp file: hi Jigar GANDHI jiga I want to convert all upper to lower. Command I tried the following command: ...
4
votes
1answer
120 views

Use paste with row-major input

I can create a file with multiple columns from a single-column input via paste like this: some_command | paste - - This works when some_command produces the data in column-major format. In other ...
1
vote
2answers
98 views

Find string then delete between two tags and repeat

This is probably a bit of a newb question, but here it goes. How can I find a string in a file, then find the first instance of a tag that precedes it and then delete everything between that tag and ...
1
vote
3answers
192 views

Splitting a string up to a specific location

I have a question and wondering how to address it with sed? How does one cut a variable up to a specific location, in my case _g? Example variable="This_is_good_g0r0s0_continues" I need to cut ...
3
votes
2answers
211 views

Sed: modify every non-first word repetition for every word in text

Can you help me to do something like that using sed? qq ab xyz ab qq aa ab Becomes: qq ab xyz +ab+ +qq+ aa +ab+
0
votes
2answers
305 views

problem with grep -f list of pattern

After searching, reading and trying possible solutions for much about grep lines with the list of patterns, hence I am writing. It is I know a very basic and have been attended in many forums. But I ...
5
votes
5answers
1k views

Use sed to print from the very first line until the line containing the last occurrence of a pattern?

I know the following commands will print up to the first occurrence of a certain pattern but will not include the later occurrences: sed -n '1,/<pattern>/p' <file> sed '/<pattern>/...
3
votes
1answer
597 views

Replace certain text occurred within curly brackets

I have sample file as follow : WEA { Direction = Input Tag = WriteEnable PortId = "A" } MEA { Direction = Input Tag = MemoryEnable PortId = "A" } CLKA { ...
2
votes
2answers
62 views

Command to work on all files in a folder and output result to single file

I have a list of files, 10 to be exact, in a recipe folder. I must determine the number of occurences of each word in all the files and then order them according to these frequencies. I can get my ...
6
votes
1answer
322 views

How to make a list generated by md5deep in alphabetical order of relative paths?

If I want make a md5sum list recursively, then I would use md5deep, but it starts to pop up some problems such as it won't generate the md5sum file in alphabetical order. For example, $ cd /media/...
2
votes
2answers
196 views

How to replace this value in bash/sed

I need help: In my file contains the following text http://mydomain.com/test.phtml http://mydomain.com/classes/main.class.phtml http://mydomain.com/scripts/filemanager/nl.phtml i need this (i ...
3
votes
2answers
248 views

Read a two-character column as two separate columns

I have a file with genotype data. The second column has both alleles for a particular genetic variant concatenated, as below. rs969931 CA 1.000 2.000 2.000 2.000 2.000 2.000 1.000 ...
2
votes
2answers
343 views

delete lines that sum to zero

I need to delete all lines in a file, if the values in all the columns are 0 (so if the sum of the row is 0). My file is like this (13 columns and 60000 rows, tab delimited) KO gene S10 S11 S12 ...
1
vote
1answer
63 views

How do I add something to the end of “}” if a pattern is matched?

If I have something that looks like this: hello {sdsdsdsds} pattern {askjdasjkdjasd} hello {siadsd} pattern {iuewer} So if pattern if matched before {*}, I need to add a ;. Output: hello {...
4
votes
3answers
1k views

How to change a set of characters between delimiters for exact line?

I am trying to write a script for changing user password in dovecot user database and I can't understand how to replace a set of characters between delimiters for exact lines with sed. Please check ...
2
votes
4answers
135 views

How do I use newlines inside () with sed?

When I want to replace, say Dot with Dot ##empty line## I do this: sed 's/Dot/Dot\ /g' Yes, a literal new line, this is the way it works on BSD sed. But, I can't do the same when the sed is ...
3
votes
1answer
417 views

grep fixed string at end of line

I need to grep for string containing special characters and make sure it is at the end of the line For example, in list: abc defg[h]ij I want to grep for defg[h]ij$ xyz 123*456 I want to grep for ...
0
votes
1answer
2k views

Is there a standalone tool which will write reports from CSV data files

I download CSV files (e.g. my bank account transactions) and would like to build reports (to print) from the data. I have searched and found several report writer libraries, but all would force me to ...
3
votes
1answer
241 views

How to find out the biggest number in many documents that contains different numbers

For example, there are some temperature data at those folders in different time. temps.txt contains the temperature number. So how can I use bash script to find out the maximum temperature? (the ...
1
vote
1answer
69 views

Add 10 to a value in large xml file

I have in a large xml file some hundred svg graphs where I want to change the <svg:g transform="matrix(1 0 0 1 59.06 88.74)"> 88.74 to 98.74 i.e to add 10 to every line that starts with &...
3
votes
7answers
15k views

Rows to column conversion of file

Suppose I have a file: File1: PAPER TEAM MANISH NISHA GARIMA JYOUTI ........etc File2 I want: PAPER TEAM MANISH NISHA GARIMA JYOUTI Rows to column conversion of File1.
4
votes
1answer
1k views

extracting a subset of lines from a large text file

I have a large file with 9 columns, tab delimited. This file is about 39MB, with about 250,000 lines. The last column, column 9, has information like this (note that this is all one column - thie ...
4
votes
8answers
646 views

Remove everything that is not a 4 character number

I need to remove everything that is not a 4 character number such as 9838, 6738, 1337 or 1889. I though that this command would work: sed 's/....[^0-9]//g' . Means any character in regex, and [^0-...
1
vote
3answers
476 views

sort CSV by number of column in rows?

I have multiple CSV files with many records. The total number of rows should be 134. I have many files, and each rows has its own number of columns (from 15 to 200). I need to sort them according to ...
0
votes
3answers
849 views

How to print only certain characters?

I have to translate the permissions of a file this way: r ► c w ► s x ► e Then, I must extract the group of characters which are related to owner group. Last step is to print that group of ...
76
votes
3answers
9k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
1
vote
1answer
393 views

sed command to print all line starting from and end to specific words present in a file

Suppose I have one file File1 consist two Words File1 : PAPER TEAM Now i need to print all the lines of a file M starting from word PAPER end to word Team sed command to print between two words ...
7
votes
7answers
1k views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
3
votes
4answers
2k views

How to use sed to selectively remove leading zeros

How can I substitute/remove 0's in this, using sed: 2001:0db8:03cd:0000:0000:ef45:0006:0123 to make it looks like this: 2001:db8:3cd:::ef45:6:123 There are 8 pieces of 4's divided by : in the ...
3
votes
1answer
344 views

Sorting lists when there are no leading zeros

I would like to sort this numerically and the sort command keys off the first character. CPL_V11.01.00_1 CPL_V11.01.00_10 CPL_V11.01.00_2 CPL_V11.01.00_3 CPL_V11.01.00_35 CPL_V11.01.00_36 CPL_V11.01....
0
votes
1answer
87 views

No output from grep -P 'com\K\w+'

I have a source file saved off the internet which has various links I want to pull data from. For example https://blahblah.com/uniquetext I'm using grep -oP 'com\K\w+' source.txt > list.txt ...
1
vote
2answers
87 views

Oneliner which prints groups and their numbers which start with a '1'?

I have to write an oneliner which print system groups and their identifiers, for all groups whose identifiers start with '1'. For example, the result of cat command on /etc/group is this: I must ...
0
votes
3answers
1k views

Group by and sum in shell script without awk

I have a file like: $ cat input.csv 201,100 201,300 300,100 300,500 100,400 I want to add the values in column 2 which has same value in column 1. Expected output is as follows: $ cat output.csv ...
3
votes
1answer
148 views

How do I use awk to create an index of words in file?

I have a problem with my homework I need to create a shell script using awk. The script will output only the words in the file in an alphabetical order, and after the word is : after that a space then ...
1
vote
3answers
307 views

Merging multiple rows into a single one using awk

I'm working with a very large data set in the format: chr1 45162 . . C T chr1 45162 . . C T chr1 45162 A . . T chr1 45162 . . C T chr1 45257 A . . T chr1 45257 A . . T chr1 45413 A . . T chr1 46844 ...
0
votes
1answer
181 views

How to Trim below highlighted lines from Shell output

How to Trim below highlighted lines from Shell output. Source shell program is below the output. $ ./checkhome.sh The Oracle base remains unchanged with value /opt/oracle Connecting ...
3
votes
2answers
151 views

How to remove specific keys from a wordlist?

I have generated a wordlist of keys starting from 03000000000 and ends at 03999999999 through crunch. So what I want is to delete a specific number of keys from it, for example, I want to delete keys ...
4
votes
3answers
598 views

Parsing Log Files with sed -e. Need to count unique class names

I have a file, let's call it filename.log, in it I have something like this (2014-11-18 14:09:21,766), , xxxxxx.local, EventSystem, DEBUG FtpsFile delay secs is 5 [pool-3-thread-7] (2014-11-18 14:09:...