Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

4
votes
8answers
257 views

Separate the element after every 2nd ',' and push into next row in bash

An example of my file looks like this: 201012,720,201011,119,201710,16 Output I want: 201012,720 201011,119 201710,16
2
votes
2answers
61 views

How to remove duplicate values based on multiple columns

I have a file tmp.log with fields like description ID valueA valueB valueC xxx x 1 1 1 yyy y 3 100 23 zzz z 0 0 0 aaa a 4 ...
0
votes
2answers
25 views

bash - split by single character and match found items by string

tl;dr: Considering following input, which is some kind of a "list" of devices from nodes, I want to get the IP addresses, but only from devices named 'router.' (not router1., nor gw. or others): gw....
0
votes
1answer
32 views

List lines that are the same in the second part but differ on the first part

Suppose that you have a big file with lines of this format: A, B, C, D (each line has 4 parts comma separated). I need a list of all the lines which have common fourth part (D is the same across those ...
1
vote
3answers
30 views

extract `.sh` scripts from a string and reformat them

I have a string like : %group1 ALL=(ALL) /ora/crs/oracle/product/crs/bin/gsdctl *,/ora/crs/oracle/product/crs/bin/vipca,/ora/app/oracle/ora/Root.sh,/ora/crs/oracle/product/crs/root.sh,/ora/crs/oracle/...
2
votes
4answers
59 views

Parse a csv file, replace separator, ignoring the content of fields between “ ”

I have a file with ; separators ; my application parses the data, based on those separators. String fields are between "" and in columns, I might have the separator as text. I would like to replace ...
1
vote
1answer
33 views

How to move lines containing specific text to top of file in same order as code executed?

How to move lines containing specific text to top of file in same order as code executed? Here is the code I have: printf 'g?_001\|_002\|_003\|_004\|_005?m0\nwq\n' | ed -s target_file ( credit ...
1
vote
3answers
64 views

How to move lines containing specific strings to top of file?

How can I move lines containing specific strings (ie. 005, and 007) to the top of the target file? Contents of target file: XXX_008 XXX_001 XXX_002 XXX_009 XXX_003 XXX_006 XXX_005 ----> located ...
1
vote
2answers
43 views

find n most frequent words in txt file using the terminal but with the cat, “grep .” and tr -d “/r” commands

I would like to find n most common words in a text file but with the following conditions: cat to get the file. grep . to get the words. tr -d '/r' to get rid of new lines. I've managed to get ...
3
votes
3answers
110 views

Repairing the headers of phylip bioinformatics files to accurately reflect the updated number of samples in the file(s)

I have a dataset that I am working with made up of phylip files that I have been editing. Phylip format is a bioinformatics format that contains as a header the number of samples and the sequence ...
-1
votes
1answer
27 views

How to remove leading numbers and whitespace from a document with python

I have long file with the following format. 123.I am biniyam sdfjhjkhkjsfsd \n sdfhhkjhkjhksdfl \n 14.I am in cybersecurity \n fdkjskdlfnmnjhuksnfe,h \n 872.I love soccer. \n I wanted to ...
0
votes
2answers
42 views

How to remove repetition of the content of file by column?

I have a file containing the following content: WP_021815421.1 OG5_132922 deth|YP_182312 3 -07 42.35 81.7 WP_021815427.1 OG5_128110 rbal|NP_868155 1 -55 50.72 53.2 WP_012456834.1 ...
0
votes
2answers
45 views

Rearranging columns in UNIX/Linux

I have to move the unique key which is in 3rd column of each record to first column. Now based on this key each record has different total columns in it (means # of fields) File content is 10,,FH,...
2
votes
2answers
94 views

How to insert a name of the file as a header of a file?

Example: File name: ENSG00000000003 ENSG00000000003 43120.829491094 ENSG00000000005 39604.4956791524 ENSG00000000419 7645.05624570546 ENSG00000000457 2157.49855156382 ENSG00000000460 3317.98417717746 ...
5
votes
5answers
380 views

Extract the characters before last colon

I need to extract the characters before last colon : and also remove the square brackets [] in the last line. My file structure is 256.XXX.XXX.X:20234 214.XXX.XXX.X:7249 [2200:XXXX:XXXX:XXX:XXXX:XXXX:...
1
vote
2answers
32 views

How to add a new column in between existing columns?

File1.txt: 1 Mark USA 2 Amir UK 3 Pant USA 4 Veer IND I have to add a new column 657890 768908 435678 342343 in 3rd position. So Output must be, Output.txt 1 Mark 657890 USA 2 Amir 768908 UK 3 ...
1
vote
2answers
51 views

Bash - print first line of textfile (csv) bold

I have a text (csv) file like this: Date:;Time:;Value:; 20181202;112044;38274658392; 20181207;121356;94843726283; 20181221;221012;93938272189; 20181230;071234;93736473783; 20190107;011022;27339393022;...
1
vote
1answer
66 views

Improve performance when using “system” call (shell escape) processing large files in awk

I have an awk script that processes very large files that look something like this: K1353 SF3987.7PD833391.4 KARE K1353 SF3987.2KD832231.4 MEAKE K1332 IF4987.7RP832231.2 LEAOS K1329 SF2787.7KD362619....
1
vote
2answers
132 views

Rearrange columns separated by : in LINUX [closed]

The question is how to rearrange the column along with values in desired order. input "a":"val1","c":"val2","b":"val3","d":"val4" "a":"val1","b":[],"c":"val3","d":"val4" "a":"val1","d":["val2","...
0
votes
2answers
35 views

how to fatch fasta sequences if header line match from another file

I have a file of lines of headers (file 1) and another file is sequences in fasta format (file 2). I want grep fasta sequences if header line from file 1 match from file 2. Example: File 1: >sp|...
0
votes
2answers
40 views

Remove a string from a particular line

I need to build a script that adds comma separated values into a particular line in a file. Say I have a file with contents: DL1 : abc@kk.com,cde@kk.com,xyz@kk.com DL2 : hjk@kk.com,nbv@kk.com,tuy@kk....
1
vote
2answers
50 views

replace header in a file with list of lines in another file

I have a fasta file contained ~28000 sequence. I want to replace header of these sequences by a list of lines in another file. Example: File 1: sp|B7UM99|TIR_ECO27 MPIGNLGNNVNGNHLIPPAPP..... sp|...
0
votes
3answers
40 views

Add data to a particular line in a file

I need to build a script that adds comma separated values into a particular line in a file. Say I have a file with contents: DL1 : abc@kk.com,cde@kk.com,xyz@kk.com DL2 : hjk@kk.com,nbv@kk.com,tuy@kk....
1
vote
2answers
48 views

how can I merge multiple column in one column and separated by '-'?

I have a table as below: 1 10 15 2 2 25 1 10 26 I like to merge them and make a new column in linux, like below: 1 10 15 1:10-15 2 2 25 2:2-25 1 10 26 1:10-26
10
votes
8answers
2k views

How to select certain lines (n, n+4, n+8, n+12…) from the file?

Input: 1 hgh h2b h4h 2 ok koko lkopk 3 uh ju nfjvn 4 Expected output: 1 2 3 4 So, I need to have only 1st, 5th, 9th, 13th value of the file in the output file. How to do this?
0
votes
1answer
31 views

Using Sed in a for loop with a regular expression

I am trying to modify a json file using a for loop to loop over a list of sample names and use sed to fix those sample names. Right now a subset of the input file is; "WA82671", "WA8267-...
4
votes
3answers
146 views

Add Columns to .csv from Multiple Files

I'm trying to add columns to a csv file from multiple files. This is what I have tried so far, but keep ending up with either an empty file or just the column from the last file. while read i; do ...
0
votes
2answers
37 views

String to replace in shell upto certain context

I want to replace string in file using sed/grep but the string is mingled with another strings Example: sed -i 's/php/vinu/g' test.yml # to replace string php with vinu test.yml php phpehealth ...
1
vote
2answers
22 views

Replace string in specific line at specific position of fixed-length file

I have a fixed-length data file where records are tagged (first 6 characters have a certain string that identifies the record) and I need to replace text from a specific line and a specific position ...
-5
votes
2answers
92 views

I have Unix scripting question

Input---- System:root,bin,user,saaa Displayed output---- System,root System,bin System,user System,says How to get this output??
0
votes
1answer
52 views

Insert characters at the middle and the end of a line with specific line length

Currently I am working on a folder of files, and each file have some lines as follows: abcde fghij abcde fghij jklmn pqrst ..... ..... These lines have a specific line length 43. At the ...
0
votes
0answers
49 views

Find duplicated columns in file

I can search a file for unique columns and it works with one file. But how do I search multiple files? awk -F"|&" 'x[$1]++ { print $1 " is duplicated"}' /home/pi/tmpdata/f1.txt This howeever ...
1
vote
1answer
49 views

Using Awk to identify and reverse word bigrams in text

I'm an avid fan of anime, and typically watch it subtitled. Sometimes the subbers will order the names as if they were Western, putting the given name before the family name. I really don't like this, ...
0
votes
2answers
80 views

How to restrict awk number of fields?

Below is my test file. aa, bb, cc dd, ee ff, gg, hh Is it possible to make awk think we only have 2 fields for each record and the record separator is the first ,? So in my example, the $2 for each ...
4
votes
3answers
169 views

How to use awk to extract data from a file based on the content of another file?

I have two files. One file includes structured data and be low is a sample. article 1 title article 1 body line 1 article 1 body line 2 +++ article 2 title article 2 body line 1 article 2 body line 2 ...
0
votes
0answers
16 views

I want to update the 4th column of file1.csv by referring 4th column of file2.csv [duplicate]

I have 2 csv files: file1.csv 20190319,006911,3939,1 20190319,006989,5107,1 20190319,007004,4242,1 20190319,007661,5107,1 20190319,014551,9881,1 20190319,026074,5107,1 20190319,039097,3331,0 ...
0
votes
1answer
23 views

Extract some portion from line with regular expression

I have file which consist lines like : 50.57.61.8 - - [04/Apr/2019:09:52:01 -0400] "GET /tracker.php?q=1&p=2 HTTP/1.1" 302 20 "-" "Rackspace Monitoring/1.1 (https://monitoring.api.rackspacecloud....
0
votes
2answers
59 views

How can I display unique words contained in a Bash string? [closed]

The contents of the words file: a-b-c c-b-a c-a-b d-a-x‬ I would like the output only: a-b-c d-a-x because the following: c-b-a c-a-b have those combinations from a-b-c
1
vote
3answers
33 views

I want to empty one column, but I want to keep the header name in unix

I am using below command to empty the column but I want to keep the header row, how can I do it in the same command? awk -F "," '{$10="";}1' OFS="," InputFile.csv > test.csv
2
votes
2answers
91 views

Two words with different last letter (Arabic)

I made dictionary file for Arabic to be used in LibreOffice and LyX. It contains over 2.7 million Arabic words. Sometimes, we can write the word with ه, and sometimes with ة if it's at the end of ...
0
votes
2answers
39 views

Remove Multiple TABS in between data

I have a tab delimited file, but in a few records, I see multiple TABS in between data Sample Data: M^I^I^I^IJaghoub (^I is tab delimiter in cat -vet output) Required Output: MJaghoub I have used ...
1
vote
2answers
42 views

Merging two files using matching characters

File1: X X P X N X Q File2: P 1 N 5 Q 0 Desired output: X 0 X 0 P 1 X 0 N 5 X 0 Q 0 I have tried a lot of ways using merge command on bash. I couldn't make it work.
7
votes
7answers
932 views

Copy previous line to current line from text file

I have a file with lines, just like this: A B C I want to create a duplicate file in bash that contains each line merged with the copy of next line, like: A;B B;C C;
2
votes
1answer
72 views

Remove deduplicates in a very large file()

In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4. so in the output there is only one row with a, one row with ...
0
votes
0answers
41 views

Want to replace the value of 4th column of data2.csv with the 4th column of other csv file data1.csv where 3rd and 4th column are same in both files

data1.csv 20180318,2121,111,0 20180318,3131,111,0 20180318,4141,111,1 20180318,5141,111,1 20180318,6161,111,0 20180318,8181,111,1 20180318,9191,111,1 20180318,9999,222,1 ...
1
vote
3answers
66 views

Replace the last alphanumeric with a random char of same type [closed]

I have a simple list of alphanumeric machine names in a file, and I want to change the last letter or number, but I want it to match character type -- so if the last character is a digit, it should be ...
0
votes
0answers
41 views

In vim in binary mode what is a ^K character?

When I open a file with vim -b it displays ^K sometimes in the text, which cannot be removed by dos2unix. It prevents dos2unix from processing the file:dos2unix: Binary symbol 0x0B found at line 628 ...
0
votes
3answers
47 views

Unintended line breaks in csv file

I have a csv file with some unintended line breaks. The file looks more or less like this: col1; col2; col3; col4 1a; 1b; 1c; 1d 2a; 2b; 2c ;2d # this should be ...
0
votes
6answers
114 views

Split string using multiple delimiter

I have a text file which contains below string and I want to convert this into a CSV file. I want to split below string: location = /example/url/newsite/redirect {return 301 https://example.com/fr;} ...
6
votes
8answers
345 views

Copy line and insert it in a new position with sed or awk

I have this file: a b c d e f I want the last line to be copied to before the first line, so I'll get the following result: f a b c d e f How can I do it in sed or awk?