Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

-1
votes
1answer
494 views

How to print gmail inbox messages

I want to print inbox messages one by one (like cat someFile) so that I can parse the message contents. something like printGmail InboxMesg1 | grep some | sed some Using curl I can access the ...
4
votes
4answers
81 views

Remove a character between 2 known strings

I have a data set like below: \"XXX \ START sapiodj \\" aj \d 2387 END hddo\" START bbcc \\" END ss My Requirement: I want to remove all occurrences of backslash\ and double quotes " between ...
5
votes
3answers
94 views

Replace uppercase to lowercase except first letter of each word in specific line

I want to replace names that I received in uppercase letters only. The names are mixed with other information which should stay as it is. AUTH: at the beginning of the line identifies it as a line ...
5
votes
6answers
4k views

How to append data of multiple files into single file?

For example: I have 01.txt and 02.txt files and need to append data of these files into new file new.txt. It should append data based on 01.txt followed by 02.txt files. Before this I have to delete ...
3
votes
4answers
130 views

tab separation of the file

I have a file with seq likes this ATCGTTTCTCAGCCTTTTGGCAAGACCAAGTGTAGTATCTGTTCTTATCAGATATTGGTATCTGCAGGCACTCCTGGAACCAAT ...
0
votes
2answers
38 views

Extract multiple instances of text between two strings [on hold]

I'd like to extract text between two strings in a text file that contains Spotify chart results Text file extract: ...
1
vote
1answer
27 views

using bash script to sort text in a file

I would like to sort the output of multipath -ll into a csv type of file using bash script. For example # multipath -ll would give the following output 360060e80056fc30000006fc30000513c dm-41 ...
6
votes
3answers
171 views

sed with multiple expression for in-place arguement

I am trying to replace multiple words in file by using sed -i #expression1 #expression2 file Something 123 item1 Something 456 item2 Something 768 item3 Something 353 item4 Output (Desired) ...
15
votes
2answers
10k views
+100

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
5
votes
2answers
47 views

Merging/Combining 2 text files according to numeric field

I have 2 text files file_1: 14595|Age 35|Salary xx|Position ax|2013|Info 1|Info 2|Info 3|Info 4|Info 5 14649|Age 30|Salary xx|Position az|2015|Info 1|Info 2|Info 3|Info 4|Info 5 file_2: ...
6
votes
4answers
153 views

Need to move the last line of the file to second line of the same file

I have a file 'test' in linux with below data. aaaaaa bbbbbb cccccc dddddd eeeeee I need to cut the last line of this file and put it in the second position. It must look like below. aaaaaa eeeeee ...
0
votes
1answer
50 views

skip line that doesn't contain perticular character eg .colon

I am reading lines from the file in while loop. while read line do #process line done I want to skip the line that does not contain colon. Any help?
0
votes
2answers
888 views

How to get the value from rows and columns from text file in bash?

I am working on a bash project. I need to get all the values (one at a time because I will be using them somewhere else) from a text file. The text file I will be getting is structured like this: ...
7
votes
5answers
429 views

List out strings which are substrings of other strings in the list

I have a list of names like so: dog_bone dog_collar dragon cool_dragon lion lion_trainer dog I need to extract out names that appear in other names like so: dragon lion dog I looked through the ...
4
votes
4answers
363 views

Convert key=value blocks to CSV

I am trying to transpose a content of a file into another. I am new to shell scripting any help would be really appreciated Input file Test.txt: HLRSN = 3 IMSI = 404212109727229 KIVALUE = ...
32
votes
3answers
2k views

Why is using a shell loop to process text considered bad practice?

Is using a while loop to process text generally considered bad practice in POSIX shells? As Stéphane Chazelas pointed out, some of the reasons for not using shell loop are conceptual, reliability, ...
3
votes
1answer
59 views

sed repeating the last line in the stream

I have code in a script similar to this... smbconffile="/etc/samba/smb.conf" sed -i 's/.*[\[CMI\]]/\[CMI\$\]/' $smbconffile && echo "Success" || "Failed" sed -i 's/.*[\[LOCAL\]]/\[LOCAL\$\]/' ...
0
votes
2answers
36 views

Take the value of another column if the query column is between two column values

So i have a file with column that has information on salary (eg. 2674 dollars) - indi salary sam 2674 john 6375 max 9547 and another file with the scale of the salary ...
-2
votes
1answer
43 views

Output a colon delimited file to another file [on hold]

I have the following example file: root:server1:2014-12-12:x:Open:adm bin daemon lp mail other root sys users sectmp:server1:2014-12-12:x:Open:users smbnull:server1:2014-12-12:x:Locked:ids ...
2
votes
5answers
2k views

Sum the values in a column except the header

I have a file as given below -------------------------------------------------------------- Name_Customer Item_Purchased Item_Amount Credit ...
0
votes
3answers
101 views

Shell script to add or replace block of text?

I'm looking for a shell script (Bash) that adds or replaces copyright text at the beginning of the file (for .c and .h files). It should search and add or replace the copyright text block for all the ...
2
votes
1answer
33 views

Cut and Paste Commands

So I have: $ cat fruits 2 bananas 3 cherries 4 figs 5 dates 6 elderberries 7 apples 8 grapes and 1 $ cat prices 2 2.18 3 4.11 4 1.69 5 4.52 6 1.73 7 1.01 8 1.09 Every line from 'fruits' ...
0
votes
4answers
786 views

How to get data count between two strings with multiple occurences in the same file

I have the data in my file as START-OF-FIELDS ID NAME DEPT END-OF-FIELDS START-OF-DATA 1|joy|cs 2|sam|ec END-OF-DATA START-OF-FIELDS ID NAME DOB DEPT ADDRESS END-OF-FIELDS START-OF-DATA ...
4
votes
7answers
684 views

Need some sed magic: Moving marked lines to the beginning of a file

I find myself in dire need of some sed magic (I really need to sit myself down and learn this). I have a file with lots of lines. After reviewing it, I've marked some lines by adding a star (*) to ...
1
vote
3answers
52 views

Calculate and round numbers up in bash

I'm struggling with a slight variation on: How to calculate by taking first field and How to round floating point numbers in shell? I have a file that looks like (space between columns): 1533 ...
2
votes
4answers
77 views

How do I use newlines inside () with sed?

When I want to replace, say Dot with Dot ##empty line## I do this: sed 's/Dot/Dot\ /g' Yes, a literal new line, this is the way it works on BSD sed. But, I can't do the same when the sed is ...
0
votes
1answer
55 views

Removing contents from a file

I have a file like this >gene1*ENSG24 CTTGGGGGGCTGGGGGCCAGGTGAAAGGGAAATGGAGGGCAGCACCCGCG AGCCCTCATTGCCTATAGTGGTTTCCATGGCGATCATGTAAGAGTCAATG TCGTCATTGGCAAAGTCGTCCGGGTGGGGTGTGCTGTAGGCAGAATCGGA ...
0
votes
2answers
52 views

Command to see if a word is in a list of files and output how many files it is in

There is a similar question to mine already but mine has a slight difference. I want to see how many files contain each word. When a file has a word then it increments the counter and then continues ...
4
votes
6answers
11k views

bash - replace space with new line

How can I replace spaces with new lines on an input like: /path/to/file /path/to/file2 /path/to/file3 /path/to/file4 /path/to/file5 etc... To obtain the following: /path/to/file /path/to/file2 ...
13
votes
5answers
427 views

non-line-oriented tool for string replacement?

I recently asked a question about how to remove the newline character if it occurs after another specific character. Unix text-processing tools are very powerful, but almost all of them deal with ...
1
vote
2answers
46 views

Extracting list of patterns which are output of another command

I can extract list of patterns using following command, fgrep -A 1 -f patternlist.txt filename.fasta but, is there a way I can extract without creating another file (patternlist.txt in this case) ...
4
votes
4answers
13k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
4
votes
3answers
84 views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
1
vote
3answers
164 views

Replace a string and replace it with a part of itself

I am cleaning up my data that is stored in text files. Each line starts with a category label followed by the actual data that I want to clean up. There are many text files in different subfolders, so ...
1
vote
5answers
98 views

How to get the unique count of a particular part of a string

I have a set of data in a file. psf7433-nlhrms unit7433-nobody unit7333-opera bpx7333-operations app7333-osm unit7330-partners psf7331-pdesmond unit7333-projm mnp7330-redirect unit7333-retailbanking ...
0
votes
2answers
30 views

output a variable value using echo command [duplicate]

I want to output number of lines in text file.So I am using echo command but I am getting filename along with number of lines when I run the following code: echo "$page" > t total="$(wc -l t)" ...
1
vote
7answers
104 views

repeat a line, splitting one field

I have a tab delimited file in which in every line I have this: K00001;K00004;K00008 0 0 34 0 0 0 0 0 0 0 0 0 0 0 0 0 36 0 0 52 0 0 0 6 0 I would ...
3
votes
2answers
56 views

How do I efficiently select specific line numbers from a list of records?

I want to take a list of numbers representing line numbers of a source file I want to filter and filter those lines from the source file. How I can build a unix pipeline to extract these lines from ...
4
votes
4answers
5k views

Concatenate multiple files with same header

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other ...
2
votes
5answers
10k views

awk-Printing column value without new line and adding comma

input.txt EN1 EN2 EN3 EN4 EN5 output EN1,EN2,EN3,EN4,EN5 I have tried awk.But it is not printing with comma awk 'BEGIN { OFS = ","} { printf $1}' input.txt I have GNU Awk 4.0.0 version
8
votes
5answers
463 views

Convert file contents to lower case

I have temp file with some lower-case and upper-case contents. Input Contents of my temp file: hi Jigar GANDHI jiga I want to convert all upper to lower. Command I tried the following command: ...
3
votes
10answers
626 views

Finding the maximum of the values in a file

I have a file which contains several numbers, as follows. 21 12 33 35 21 12 33 44 52 63 14 12 23 34 11 12 13 53 1 12 43 33 44 How can I find the maximum of these values, which is 63? I wanted to ...
0
votes
3answers
2k views

compare two files based on a column and print it

I have two big files of 400,000 lines. I want to compare the column 1 of the second file with column 1 of first file recursively. If they match I would like to print the whole line. It is a sorted ...
1
vote
3answers
125 views

grep only matching word

I have a temp file, and I want to grep the only word that matches the pattern instead of whole word. i tried grep -o <pattern> file but its not giving me desired output Input xi29 vddf vss ...
6
votes
4answers
297 views

How to split text between separator into multiple files?

I have a text file contaning the following: "random textA" "random random textB" The separator is " How can I split the containt into multiple file as follow using a bash command? File 1 : ...
-1
votes
2answers
31 views

Extract 2 fields from 2 different lines

I have the below lines in my file # Employee Id : ABC123, # Employee Name: Vivek # Employee Type: Temp Join Date: 31-jan-2014 Buz Code: 2098 # Employee Comments: # ...
3
votes
3answers
86 views

Comment (automatically) a LaTeX source file to make it more readable

I am looking for a simple way to do the following : Input : \begin{document} ... \section{} ... \subsection{} ... \subsubsection{} ... \section{} ... \end{document} Output : \begin{document} ... ...
1
vote
2answers
53 views

How to get portion of lines from all .txt files in a directory?

I have 5,000 text files of journal article citations. I am trying to extract only the abstract portion. Meaning that I want to keep the same text document and delete all the other text except for ...
2
votes
2answers
94 views

Remove escape characters using sed

I have input coming from pipe which contains special escape characters. For illustration, the stream looks like this: printf "foo.bar\033[00m" > file cat file How can I remove the trailing ...
3
votes
1answer
23 views

Replacing " with pretty quotes on a lot of files

I have a lot of UTF-8 encoded files with words quoted like this "word" and I want to replace the quotes with “word” (the first quotes are upside down and the second is normal) Because the ...