Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
4
votes
8answers
257 views
Separate the element after every 2nd ',' and push into next row in bash
An example of my file looks like this:
201012,720,201011,119,201710,16
Output I want:
201012,720
201011,119
201710,16
2
votes
2answers
61 views
How to remove duplicate values based on multiple columns
I have a file tmp.log with fields like
description ID valueA valueB valueC
xxx x 1 1 1
yyy y 3 100 23
zzz z 0 0 0
aaa a 4 ...
0
votes
2answers
25 views
bash - split by single character and match found items by string
tl;dr:
Considering following input, which is some kind of a "list" of devices from nodes, I want to get the IP addresses, but only from devices named 'router.' (not router1., nor gw. or others):
gw....
0
votes
1answer
32 views
List lines that are the same in the second part but differ on the first part
Suppose that you have a big file with lines of this format: A, B, C, D (each line has 4 parts comma separated).
I need a list of all the lines which have common fourth part (D is the same across those ...
1
vote
3answers
30 views
extract `.sh` scripts from a string and reformat them
I have a string like :
%group1 ALL=(ALL) /ora/crs/oracle/product/crs/bin/gsdctl *,/ora/crs/oracle/product/crs/bin/vipca,/ora/app/oracle/ora/Root.sh,/ora/crs/oracle/product/crs/root.sh,/ora/crs/oracle/...
2
votes
4answers
59 views
Parse a csv file, replace separator, ignoring the content of fields between “ ”
I have a file with ; separators ; my application parses the data, based on those separators. String fields are between "" and in columns, I might have the separator as text.
I would like to replace ...
1
vote
1answer
33 views
How to move lines containing specific text to top of file in same order as code executed?
How to move lines containing specific text to top of file in same order as code executed?
Here is the code I have:
printf 'g?_001\|_002\|_003\|_004\|_005?m0\nwq\n' | ed -s target_file
( credit ...
1
vote
3answers
64 views
How to move lines containing specific strings to top of file?
How can I move lines containing specific strings (ie. 005, and 007) to the top of the target file?
Contents of target file:
XXX_008
XXX_001
XXX_002
XXX_009
XXX_003
XXX_006
XXX_005 ----> located ...
1
vote
2answers
43 views
find n most frequent words in txt file using the terminal but with the cat, “grep .” and tr -d “/r” commands
I would like to find n most common words in a text file but with the following conditions:
cat to get the file.
grep . to get the words.
tr -d '/r' to get rid of new lines.
I've managed to get ...
3
votes
3answers
110 views
Repairing the headers of phylip bioinformatics files to accurately reflect the updated number of samples in the file(s)
I have a dataset that I am working with made up of phylip files that I have been editing. Phylip format is a bioinformatics format that contains as a header the number of samples and the sequence ...
-1
votes
1answer
27 views
How to remove leading numbers and whitespace from a document with python
I have long file with the following format.
123.I am biniyam sdfjhjkhkjsfsd \n
sdfhhkjhkjhksdfl \n
14.I am in cybersecurity \n
fdkjskdlfnmnjhuksnfe,h \n
872.I love soccer. \n
I wanted to ...
0
votes
2answers
42 views
How to remove repetition of the content of file by column?
I have a file containing the following content:
WP_021815421.1 OG5_132922 deth|YP_182312 3 -07 42.35 81.7
WP_021815427.1 OG5_128110 rbal|NP_868155 1 -55 50.72 53.2
WP_012456834.1 ...
0
votes
2answers
45 views
Rearranging columns in UNIX/Linux
I have to move the unique key which is in 3rd column of each record to first column. Now based on this key each record has different total columns in it (means # of fields)
File content is
10,,FH,...
2
votes
2answers
94 views
How to insert a name of the file as a header of a file?
Example: File name: ENSG00000000003
ENSG00000000003 43120.829491094
ENSG00000000005 39604.4956791524
ENSG00000000419 7645.05624570546
ENSG00000000457 2157.49855156382
ENSG00000000460 3317.98417717746
...
5
votes
5answers
380 views
Extract the characters before last colon
I need to extract the characters before last colon : and also remove the square brackets [] in the last line.
My file structure is
256.XXX.XXX.X:20234
214.XXX.XXX.X:7249
[2200:XXXX:XXXX:XXX:XXXX:XXXX:...
1
vote
2answers
32 views
How to add a new column in between existing columns?
File1.txt:
1 Mark USA
2 Amir UK
3 Pant USA
4 Veer IND
I have to add a new column
657890
768908
435678
342343
in 3rd position. So Output must be,
Output.txt
1 Mark 657890 USA
2 Amir 768908 UK
3 ...
1
vote
2answers
51 views
Bash - print first line of textfile (csv) bold
I have a text (csv) file like this:
Date:;Time:;Value:;
20181202;112044;38274658392;
20181207;121356;94843726283;
20181221;221012;93938272189;
20181230;071234;93736473783;
20190107;011022;27339393022;...
1
vote
1answer
66 views
Improve performance when using “system” call (shell escape) processing large files in awk
I have an awk script that processes very large files that look something like this:
K1353 SF3987.7PD833391.4 KARE
K1353 SF3987.2KD832231.4 MEAKE
K1332 IF4987.7RP832231.2 LEAOS
K1329 SF2787.7KD362619....
1
vote
2answers
132 views
Rearrange columns separated by : in LINUX [closed]
The question is how to rearrange the column along with values in desired order.
input
"a":"val1","c":"val2","b":"val3","d":"val4"
"a":"val1","b":[],"c":"val3","d":"val4"
"a":"val1","d":["val2","...
0
votes
2answers
35 views
how to fatch fasta sequences if header line match from another file
I have a file of lines of headers (file 1) and another file is sequences in fasta format (file 2). I want grep fasta sequences if header line from file 1 match from file 2.
Example:
File 1:
>sp|...
0
votes
2answers
40 views
Remove a string from a particular line
I need to build a script that adds comma separated values into a particular line in a file.
Say I have a file with contents:
DL1 : abc@kk.com,cde@kk.com,xyz@kk.com
DL2 : hjk@kk.com,nbv@kk.com,tuy@kk....
1
vote
2answers
50 views
replace header in a file with list of lines in another file
I have a fasta file contained ~28000 sequence. I want to replace header of these sequences by a list of lines in another file.
Example:
File 1:
sp|B7UM99|TIR_ECO27
MPIGNLGNNVNGNHLIPPAPP.....
sp|...
0
votes
3answers
40 views
Add data to a particular line in a file
I need to build a script that adds comma separated values into a particular line in a file.
Say I have a file with contents:
DL1 : abc@kk.com,cde@kk.com,xyz@kk.com
DL2 : hjk@kk.com,nbv@kk.com,tuy@kk....
1
vote
2answers
48 views
how can I merge multiple column in one column and separated by '-'?
I have a table as below:
1 10 15
2 2 25
1 10 26
I like to merge them and make a new column in linux, like below:
1 10 15 1:10-15
2 2 25 2:2-25
1 10 26 1:10-26
10
votes
8answers
2k views
How to select certain lines (n, n+4, n+8, n+12…) from the file?
Input:
1
hgh
h2b
h4h
2
ok
koko
lkopk
3
uh
ju
nfjvn
4
Expected output:
1
2
3
4
So, I need to have only 1st, 5th, 9th, 13th value of the file in the output file. How to do this?
0
votes
1answer
31 views
Using Sed in a for loop with a regular expression
I am trying to modify a json file using a for loop to loop over a list of sample names and use sed to fix those sample names. Right now a subset of the input file is;
"WA82671",
"WA8267-...
4
votes
3answers
146 views
Add Columns to .csv from Multiple Files
I'm trying to add columns to a csv file from multiple files. This is what I have tried so far, but keep ending up with either an empty file or just the column from the last file.
while read i; do
...
0
votes
2answers
37 views
String to replace in shell upto certain context
I want to replace string in file using sed/grep but the string is mingled with another strings
Example:
sed -i 's/php/vinu/g' test.yml # to replace string php with vinu
test.yml
php
phpehealth
...
1
vote
2answers
22 views
Replace string in specific line at specific position of fixed-length file
I have a fixed-length data file where records are tagged (first 6 characters have a certain string that identifies the record) and I need to replace text from a specific line and a specific position ...
-5
votes
2answers
92 views
I have Unix scripting question
Input----
System:root,bin,user,saaa
Displayed output----
System,root
System,bin
System,user
System,says
How to get this output??
0
votes
1answer
52 views
Insert characters at the middle and the end of a line with specific line length
Currently I am working on a folder of files, and each file have some lines as follows:
abcde fghij
abcde fghij
jklmn pqrst
..... .....
These lines have a specific line length 43. At the ...
0
votes
0answers
49 views
Find duplicated columns in file
I can search a file for unique columns and it works with one file. But how do I search multiple files?
awk -F"|&" 'x[$1]++ { print $1 " is duplicated"}' /home/pi/tmpdata/f1.txt
This howeever ...
1
vote
1answer
49 views
Using Awk to identify and reverse word bigrams in text
I'm an avid fan of anime, and typically watch it subtitled. Sometimes the subbers will order the names as if they were Western, putting the given name before the family name. I really don't like this, ...
0
votes
2answers
80 views
How to restrict awk number of fields?
Below is my test file.
aa, bb, cc
dd, ee
ff, gg, hh
Is it possible to make awk think we only have 2 fields for each record and the record separator is the first ,? So in my example, the $2 for each ...
4
votes
3answers
169 views
How to use awk to extract data from a file based on the content of another file?
I have two files. One file includes structured data and be low is a sample.
article 1 title
article 1 body line 1
article 1 body line 2
+++
article 2 title
article 2 body line 1
article 2 body line 2
...
0
votes
0answers
16 views
I want to update the 4th column of file1.csv by referring 4th column of file2.csv [duplicate]
I have 2 csv files:
file1.csv
20190319,006911,3939,1
20190319,006989,5107,1
20190319,007004,4242,1
20190319,007661,5107,1
20190319,014551,9881,1
20190319,026074,5107,1
20190319,039097,3331,0
...
0
votes
1answer
23 views
Extract some portion from line with regular expression
I have file which consist lines like :
50.57.61.8 - - [04/Apr/2019:09:52:01 -0400] "GET /tracker.php?q=1&p=2 HTTP/1.1" 302 20 "-" "Rackspace Monitoring/1.1 (https://monitoring.api.rackspacecloud....
0
votes
2answers
59 views
How can I display unique words contained in a Bash string? [closed]
The contents of the words file:
a-b-c
c-b-a
c-a-b
d-a-x
I would like the output only:
a-b-c
d-a-x
because the following:
c-b-a
c-a-b
have those combinations from a-b-c
1
vote
3answers
33 views
I want to empty one column, but I want to keep the header name in unix
I am using below command to empty the column but I want to keep the header row, how can I do it in the same command?
awk -F "," '{$10="";}1' OFS="," InputFile.csv > test.csv
2
votes
2answers
91 views
Two words with different last letter (Arabic)
I made dictionary file for Arabic to be used in LibreOffice and LyX. It contains over 2.7 million Arabic words.
Sometimes, we can write the word with ه, and sometimes with ة if it's at the end of ...
0
votes
2answers
39 views
Remove Multiple TABS in between data
I have a tab delimited file, but in a few records, I see multiple TABS in between data
Sample Data: M^I^I^I^IJaghoub (^I is tab delimiter in cat -vet output)
Required Output: MJaghoub
I have used ...
1
vote
2answers
42 views
Merging two files using matching characters
File1:
X
X
P
X
N
X
Q
File2:
P 1
N 5
Q 0
Desired output:
X 0
X 0
P 1
X 0
N 5
X 0
Q 0
I have tried a lot of ways using merge command on bash. I couldn't make it work.
7
votes
7answers
932 views
Copy previous line to current line from text file
I have a file with lines, just like this:
A
B
C
I want to create a duplicate file in bash that contains each line merged with the copy of next line, like:
A;B
B;C
C;
2
votes
1answer
72 views
Remove deduplicates in a very large file()
In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4.
so in the output there is only one row with a, one row with ...
0
votes
0answers
41 views
Want to replace the value of 4th column of data2.csv with the 4th column of other csv file data1.csv where 3rd and 4th column are same in both files
data1.csv
20180318,2121,111,0
20180318,3131,111,0
20180318,4141,111,1
20180318,5141,111,1
20180318,6161,111,0
20180318,8181,111,1
20180318,9191,111,1
20180318,9999,222,1
...
1
vote
3answers
66 views
Replace the last alphanumeric with a random char of same type [closed]
I have a simple list of alphanumeric machine names in a file, and I want to change the last letter or number, but I want it to match character type -- so if the last character is a digit, it should be ...
0
votes
0answers
41 views
In vim in binary mode what is a ^K character?
When I open a file with vim -b it displays ^K sometimes in the text, which cannot be removed by dos2unix. It prevents dos2unix from processing the file:dos2unix: Binary symbol 0x0B found at line 628
...
0
votes
3answers
47 views
Unintended line breaks in csv file
I have a csv file with some unintended line breaks.
The file looks more or less like this:
col1; col2; col3; col4
1a; 1b; 1c; 1d
2a; 2b; 2c
;2d # this should be ...
0
votes
6answers
114 views
Split string using multiple delimiter
I have a text file which contains below string and I want to convert this into a CSV file.
I want to split below string:
location = /example/url/newsite/redirect {return 301 https://example.com/fr;}
...
6
votes
8answers
345 views
Copy line and insert it in a new position with sed or awk
I have this file:
a
b
c
d
e
f
I want the last line to be copied to before the first line, so I'll get the following result:
f
a
b
c
d
e
f
How can I do it in sed or awk?
