Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

0
votes
3answers
78 views

Strip the file base name, keep the directory part

I have a string which looks something like this dir/subdir/othersubdir/file.txt But it can also look like dir/file.txt or dir/subdir/file.txt Now, I am making a script where $P is the full ...
4
votes
3answers
203 views

Copy a specific percentage of each file in a directory to a new file

For example, we have N files (file1, file2, file3 ...) We need first 20% of them, the result directory should be like (file1_20, file2_20, file3_20 ...). I was thinking use wc to get the lines of ...
2
votes
1answer
84 views

Specify lines in awk?

Im going to use this awk script to delete duplicate lines from my text file: awk '!x[$0]++' However, I want to specify which lines that they are to be run on. There could be 10000 duplicates, but ...
0
votes
2answers
242 views

Shell script to add or replace block of text?

I'm looking for a shell script (Bash) that adds or replaces copyright text at the beginning of the file (for .c and .h files). It should search and add or replace the copyright text block for all the ...
2
votes
1answer
52 views

sed: append line and number output

In Linux, say one wants to add the following line before line number five (thus line nuber 5 becomes line number 6) and wants to have all the lines in the output numbered throught the sed 'tool' = ...
2
votes
7answers
384 views

How to print lines with blank 5th field in CSV

I have to check if the fifth field is empty in a CSV file. This is my file: 1,abc,543,87,DATA,fsg; 1,abc,543,87,,fsg; 1,abc,543,87,DATA,fsg; 1,abc,543,88,,fsg; 1,abc,543,,DATA,fsg; As you can ...
8
votes
5answers
989 views

Convert file contents to lower case

I have temp file with some lower-case and upper-case contents. Input Contents of my temp file: hi Jigar GANDHI jiga I want to convert all upper to lower. Command I tried the following command: ...
3
votes
1answer
57 views

Use paste with row-major input

I can create a file with multiple columns from a single-column input via paste like this: some_command | paste - - This works when some_command produces the data in column-major format. In other ...
1
vote
2answers
45 views

Find string then delete between two tags and repeat

This is probably a bit of a newb question, but here it goes. How can I find a string in a file, then find the first instance of a tag that precedes it and then delete everything between that tag and ...
1
vote
3answers
55 views

Splitting a string up to a specific location

I have a question and wondering how to address it with sed? How does one cut a variable up to a specific location, in my case _g? Example variable="This_is_good_g0r0s0_continues" I need to cut ...
3
votes
2answers
162 views

Sed: modify every non-first word repetition for every word in text

Can you help me to do something like that using sed? qq ab xyz ab qq aa ab Becomes: qq ab xyz +ab+ +qq+ aa +ab+
0
votes
2answers
83 views

problem with grep -f list of pattern

After searching, reading and trying possible solutions for much about grep lines with the list of patterns, hence I am writing. It is I know a very basic and have been attended in many forums. But I ...
4
votes
5answers
175 views

Use sed to print from the very first line until the line containing the last occurrence of a pattern?

I know the following commands will print up to the first occurrence of a certain pattern but will not include the later occurrences: sed -n '1,/<pattern>/p' <file> sed ...
2
votes
1answer
166 views

Replace certain text occurred within curly brackets

I have sample file as follow : WEA { Direction = Input Tag = WriteEnable PortId = "A" } MEA { Direction = Input Tag = MemoryEnable PortId = "A" } CLKA { ...
2
votes
2answers
47 views

Command to work on all files in a folder and output result to single file

I have a list of files, 10 to be exact, in a recipe folder. I must determine the number of occurences of each word in all the files and then order them according to these frequencies. I can get my ...
5
votes
1answer
153 views

How to make a list generated by md5deep in alphabetical order of relative paths?

If I want make a md5sum list recursively, then I would use md5deep, but it starts to pop up some problems such as it won't generate the md5sum file in alphabetical order. For example, $ cd ...
2
votes
2answers
132 views

How to replace this value in bash/sed

I need help: In my file contains the following text http://mydomain.com/test.phtml http://mydomain.com/classes/main.class.phtml http://mydomain.com/scripts/filemanager/nl.phtml i need this (i ...
3
votes
2answers
88 views

Read a two-character column as two separate columns

I have a file with genotype data. The second column has both alleles for a particular genetic variant concatenated, as below. rs969931 CA 1.000 2.000 2.000 2.000 2.000 2.000 1.000 ...
2
votes
2answers
97 views

delete lines that sum to zero

I need to delete all lines in a file, if the values in all the columns are 0 (so if the sum of the row is 0). My file is like this (13 columns and 60000 rows, tab delimited) KO gene S10 S11 S12 ...
1
vote
1answer
50 views

How do I add something to the end of “}” if a pattern is matched?

If I have something that looks like this: hello {sdsdsdsds} pattern {askjdasjkdjasd} hello {siadsd} pattern {iuewer} So if pattern if matched before {*}, I need to add a ;. Output: hello ...
4
votes
3answers
294 views

How to change a set of characters between delimiters for exact line?

I am trying to write a script for changing user password in dovecot user database and I can't understand how to replace a set of characters between delimiters for exact lines with sed. Please check ...
2
votes
4answers
85 views

How do I use newlines inside () with sed?

When I want to replace, say Dot with Dot ##empty line## I do this: sed 's/Dot/Dot\ /g' Yes, a literal new line, this is the way it works on BSD sed. But, I can't do the same when the sed is ...
3
votes
1answer
135 views

grep fixed string at end of line

I need to grep for string containing special characters and make sure it is at the end of the line For example, in list: abc defg[h]ij I want to grep for defg[h]ij$ xyz 123*456 I want to grep for ...
0
votes
1answer
87 views

Is there a standalone tool which will write reports from CSV data files

I download CSV files (e.g. my bank account transactions) and would like to build reports (to print) from the data. I have searched and found several report writer libraries, but all would force me to ...
3
votes
1answer
60 views

How to find out the biggest number in many documents that contains different numbers

For example, there are some temperature data at those folders in different time. temps.txt contains the temperature number. So how can I use bash script to find out the maximum temperature? (the ...
1
vote
1answer
58 views

Add 10 to a value in large xml file

I have in a large xml file some hundred svg graphs where I want to change the <svg:g transform="matrix(1 0 0 1 59.06 88.74)"> 88.74 to 98.74 i.e to add 10 to every line that starts with ...
3
votes
7answers
472 views

Rows to column conversion of file

Suppose I have a file: File1: PAPER TEAM MANISH NISHA GARIMA JYOUTI ........etc File2 I want: PAPER TEAM MANISH NISHA GARIMA JYOUTI Rows to column conversion of File1.
4
votes
1answer
224 views

extracting a subset of lines from a large text file

I have a large file with 9 columns, tab delimited. This file is about 39MB, with about 250,000 lines. The last column, column 9, has information like this (note that this is all one column - thie ...
4
votes
8answers
596 views

Remove everything that is not a 4 character number

I need to remove everything that is not a 4 character number such as 9838, 6738, 1337 or 1889. I though that this command would work: sed 's/....[^0-9]//g' . Means any character in regex, and ...
1
vote
3answers
221 views

sort CSV by number of column in rows?

I have multiple CSV files with many records. The total number of rows should be 134. I have many files, and each rows has its own number of columns (from 15 to 200). I need to sort them according to ...
0
votes
3answers
463 views

How to print only certain characters?

I have to translate the permissions of a file this way: r ► c w ► s x ► e Then, I must extract the group of characters which are related to owner group. Last step is to print that group of ...
34
votes
3answers
2k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
0
votes
1answer
81 views

sed command to print all line starting from and end to specific words present in a file

Suppose I have one file File1 consist two Words File1 : PAPER TEAM Now i need to print all the lines of a file M starting from word PAPER end to word Team sed command to print between two words ...
6
votes
7answers
584 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
3
votes
4answers
400 views

How to use sed to selectively remove leading zeros

How can I substitute/remove 0's in this, using sed: 2001:0db8:03cd:0000:0000:ef45:0006:0123 to make it looks like this: 2001:db8:3cd:::ef45:6:123 There are 8 pieces of 4's divided by : in the ...
3
votes
1answer
125 views

Sorting lists when there are no leading zeros

I would like to sort this numerically and the sort command keys off the first character. CPL_V11.01.00_1 CPL_V11.01.00_10 CPL_V11.01.00_2 CPL_V11.01.00_3 CPL_V11.01.00_35 CPL_V11.01.00_36 ...
0
votes
1answer
57 views

No output from grep -P 'com\K\w+'

I have a source file saved off the internet which has various links I want to pull data from. For example https://blahblah.com/uniquetext I'm using grep -oP 'com\K\w+' source.txt > list.txt ...
1
vote
2answers
77 views

Oneliner which prints groups and their numbers which start with a '1'?

I have to write an oneliner which print system groups and their identifiers, for all groups whose identifiers start with '1'. For example, the result of cat command on /etc/group is this: I must ...
0
votes
3answers
216 views

Group by and sum in shell script without awk

I have a file like: $ cat input.csv 201,100 201,300 300,100 300,500 100,400 I want to add the values in column 2 which has same value in column 1. Expected output is as follows: $ cat output.csv ...
3
votes
1answer
84 views

How do I use awk to create an index of words in file?

I have a problem with my homework I need to create a shell script using awk. The script will output only the words in the file in an alphabetical order, and after the word is : after that a space then ...
1
vote
3answers
89 views

Merging multiple rows into a single one using awk

I'm working with a very large data set in the format: chr1 45162 . . C T chr1 45162 . . C T chr1 45162 A . . T chr1 45162 . . C T chr1 45257 A . . T chr1 45257 A . . T chr1 45413 A . . T chr1 46844 ...
0
votes
1answer
50 views

How to Trim below highlighted lines from Shell output

How to Trim below highlighted lines from Shell output. Source shell program is below the output. $ ./checkhome.sh The Oracle base remains unchanged with value /opt/oracle Connecting ...
3
votes
2answers
72 views

How to remove specific keys from a wordlist?

I have generated a wordlist of keys starting from 03000000000 and ends at 03999999999 through crunch. So what I want is to delete a specific number of keys from it, for example, I want to delete keys ...
3
votes
3answers
66 views

Parsing Log Files with sed -e. Need to count unique class names

I have a file, let's call it filename.log, in it I have something like this (2014-11-18 14:09:21,766), , xxxxxx.local, EventSystem, DEBUG FtpsFile delay secs is 5 [pool-3-thread-7] (2014-11-18 ...
3
votes
5answers
205 views

removing extensions in a column

I have a file like this ILMN_1343291 TGTGTTGAGAGCTTCTCAGACTATCCACCTTTGGGTCGCTTTGCTGTTCG NM_001402.5 ILMN_1343295 CTTCAACAGCGACACCCACTCCTCCACCTTTGACGCTGGGGCTGGCATTG NM_002046.3 ILMN_1651209 ...
1
vote
2answers
92 views

Search for pattern and append line to another file

I have a file like this (five tab-separated columns) head allKO.txt Metabolism Carbohydrate metabolism Glycolisis K07448 Metabolism Protein metabolism protesome K02217 and I want to search for the ...
2
votes
1answer
53 views

How can I add four spaces before each lines of programs code in terminal

I usually go to stackexchange when I face programming terminal and I spend most of time trying to append four spaces before my pasted programming code. Is there any faster way to do it from a ...
0
votes
2answers
72 views

Grep matching cell of the csv file and index of that row

I have csv file and I want to see numbers that start with 0.99 and their row's index which is the first cell of that row. This is what I have so far: cat fil.csv | grep '0\\.99'| tee > (cut -d, ...
3
votes
4answers
339 views

Command to remove a portion of JSON data from each line?

I have test.json file with different lengths of rows. Some fictitious example: { a: 123, b: sd, c: x45, d: 1, e: '' } { a: 5, b: bfgg, c: x4c, d: 31, e: '' } I want to cut the whole substring after ...
3
votes
3answers
164 views

Grep for searching pattern in a file

I want to use grep to search pattern from one file in a second one. My pattern file is something like: K02217 K07448 KO8980 The file to search in is: >aai:AARI_24510 proP; proline/betaine ...