Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
-1
votes
1answer
16 views
Joining 2 files and excluding word from third file
I have three files: file1, file2 and exl. Both file1 and file2 have two columns of content. The first column is the same in both files.
I want to join the content of both files using that column. ...
1
vote
3answers
30 views
Remove all parent directories from a list of paths
I have a bash command that lists the directories in HDFS beginning with /random.
It goes as follows:
hdfs dfs -ls -R /random | grep '^d' | awk '{print $NF}'
It generates the following output:
/...
2
votes
1answer
32 views
`sed` updates the meta dates of all accessed files
I used sed -i 's/oldpat/newpat/g' ~/mydir/**/*.txt to replace some str in some text files.
However, the modified dates of all files, including those with contents unchanged, were changed to current ...
0
votes
2answers
20 views
Copy values from a file to a new file only after certain characters
I have a file with the following format:
Reading frame 3000 time 30000.000 ^MReading frame 3000 time 30000.000 mu 8.788e+00 <mu> 2.945e-03
and I want to paste the values which are ...
0
votes
1answer
33 views
cumulative table, create new table in which values in each column are sum of each previous column values [on hold]
I am dealing with script to transform table - in output table each column should be a sum of previous columns (something like cummulative table). Table got unknown number of rows and columns, first ...
1
vote
3answers
37 views
How to loop through first file and print each row in the first file next to all the rows in the second file using AWK?
I think the title may be confusing, so here's a small example of what I am trying to accomplish.
File 1:
Adam
John
Tim
File 2:
,2018-2-1301
,2018-3-1379
,2018-1-3809
,2018-2-5801
Output:
Adam,...
1
vote
2answers
48 views
remove first two bytes of a line
I have a file that has 2 bytes of binary indicator values in every row that has to be stripped off.
I tried sed, awk, cut but none of them work (though they work in vi mode):
sed 's/^.\{,2\}//' file....
0
votes
2answers
56 views
Print the lines that only with all consecutive repeated characters
I have a text file and I want to print only the lines that have all their characters are consecutively reapeated at least two times or more.
FOR EXAMPLE THE INPUT:
11
AAA
555227777
BBhh@@222
baabbb
...
0
votes
3answers
18 views
count matching open ports in a network log file
I have network log file like this one:
Nmap scan report for 192.168.1.51
Host is up.
PORT STATE SERVICE
80/tcp open http
443/tcp open https
8080/tcp open http-proxy
443/tcp ...
6
votes
5answers
734 views
Making summary of sentences
I have data and I want to summarize sentences to generate conclusions. The example below is not related to the data, but just to clarify the idea so I can replicate it.
Employee Suzie signed one time....
1
vote
1answer
43 views
How to remove pattern from line but return what remains?
If I run ip addr sho | grep 9000 it returns the interface that I need quick stats for:
eth3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
However, I'd like to ...
1
vote
3answers
58 views
extract, cut, process strings in a bash script
I have basically 4 txt files with records such as
FILE 1
localhost_access_2018-07-19.tar.gz:13
localhost_access_2018-07-20.tar.gz:17
localhost_access_2018-07-21.tar.gz:12
localhost_access_2018-07-...
0
votes
4answers
42 views
unix sed remove adapter names
I would like to pipe the output to sed and remove the below adapter names using sed and group the values with commas
ent13: 16 3979 ent12: 211 411 3978 ent11: 15 3977 ent10: 111 178 179 311 511 1719 ...
4
votes
3answers
132 views
Deleting portion of a text file and following lines using sed
I need to edit a file like the following:
auto wlx00
allow-hotplug wlx00
iface wlx000 inet dhcp
iface wlx000 inet6 auto
post-up sysctl -w net.ipv6.conf.wlx000.accept_ra=2
auto wlx000
the goal is ...
5
votes
4answers
528 views
How can I find all lines containing two specified words? [duplicate]
I need to check if two (specified) words exist on any line in a text file. There are no limits for the characters of the words. For example:
I want to find lines of a text file that contain the two ...
0
votes
1answer
23 views
how to copy columns from file 2 to file 1 when some columns matches between 2 files
I have a file
head top_candidates
25 elevation_e gene1 20 9 0.0246022994932004 5 8 10.9217937824527
30 elevation_e gene1 59 18 0.0246022994932004 7 12 15.653559774527
31 elevation_e ...
-4
votes
3answers
80 views
File Processing to transpose data [on hold]
I have a file with IDs and names applicable to them as below:
1234|abc|cde|fgh
5678|ijk|abc|lmn
9101|cde|fgh|klm
1213|klm|abc|cde
I should get the output as different files having IDs matching each ...
0
votes
2answers
45 views
how can i move lines in mulitple text files? [duplicate]
Hi assume I have a files that look like:
x
x
x
A
B
C
1
2
3
D
E
F
x
x
x
and what need the files to look like:
x
x
x
A
B
C
D
E
F
1
2
3
x
x
x
I know I could do this with a few sed -i s/search/replace/...
17
votes
3answers
1k views
Removing all non-ascii characters from a workflow (file)
How would I remove all non-ascii characters from one file? Would there be a specific command to perform this?
grep --colour='auto' -P -n'[^\x00-\x7]' /usr/local/...
I believe this finds the ...
-2
votes
2answers
48 views
Find longest stretch of pattern from centre of sequence
How to calculate the longest consecutive pattern of C/T anywhere in sequence and longest consecutive pattern of C/T from centre
CCCCTGTTGCCAAACAATGC
TTTTCCCGCCTTTGGCCTAC
...
0
votes
1answer
28 views
Extract names from File_B having overlapping intervals with File_A
Two space-delimited files:
File_A
MT 50000
groupI 7850000
groupI 7950000
groupI 9050000
groupI 21750000
groupII 8750000
groupII 10550000
groupII 16150000
groupII 20850000
groupIII 14750000
groupIII ...
-1
votes
0answers
20 views
Regular expression to match only last folder [duplicate]
I would like to get the name of the last folder in a path for use in bash script.
For example: I have a path /media/devicename/folder1/ I would like to get a variable called foldername that has in it ...
3
votes
3answers
546 views
Find how many times a certain DNA base sequence occurs in a file
The assignment is to write a bash script named “countmatches” that will display the number of times a certain sequence, such as aac, appears in a specified file. The script should expect at least two ...
1
vote
2answers
174 views
Add four lines between sentences [duplicate]
Is there an easy way to add four empty lines between each sentence.
For example, this paragraph:
This combination of vitamins and selected minerals is used to prevent or treat vitamin
deficiency ...
4
votes
2answers
66 views
removing digits from the end of lines in text file
I have this sample from a log file which is driving me crazy. i want to remove the last 8 digits but the problem there are colons in different places so I tried awk but it didn't work. this is the ...
0
votes
3answers
65 views
How to split a file into paragraphs and name the resulting pieces based on an identifier present in each paragraph
I have a big file with more than 3264880 lines. I wanted to split that file based on Two strings "BEGIN JOB" and " END JOB" and write it into multiple files and the file name should be based on a ...
0
votes
2answers
80 views
Unix script to read the file and create a file for each line
I have a file and my task is to read the file and generate a new file for each line. The name of newly generated file should come from the line content.
Example
The file has these two lines:
My ...
0
votes
1answer
14 views
Filter file to list all paths in it
How can I filter text file to remove every thing and just leave the paths in it?
For example:
dslkjcblasdjncklasdnlc /asdds/ads/adss/sdf.jpg asljdascuashn sdkm
/sdf/dsf/dsf/dfs/sdfs.txt ,nbaskhdnasm....
0
votes
3answers
34 views
Replace space with Pipe between the tags
I have a file that is coming in tags "{}". Need to convert it into a pipe format.
Here is the sample:
{Id USXXXXXX} {Code USXXXXX} {Desc abcd 6.000% 5/24}
Expected result 1:
{Id|USXXXXXX} {Code|...
1
vote
4answers
55 views
How to move a line to the previous line if a certain character “ : ” is not found? [duplicate]
I have a lot of text files of transcripts. I have cleaned it to an extent. The last bit of cleaning is the following.
I have this in certain files *.txt
Gary: I said something.
Larry: I said ...
4
votes
4answers
58 views
How to compare the 1st column of two files and if the strings match then print “true” [duplicate]
I have two files, say file1 and file2.
file1:
91 | 2018-10-11 16:07:14 | bkwdv | 188.0000 |
77 | 2018-10-12 15:05:15 | wvwef | 94.0000 |
file2:
77 | 2018-10-12 15:05:15 | wvwef | 94.0000
91 | ...
-2
votes
1answer
58 views
How to do awk to do sorting between a txt file and a csv file? [closed]
Here is the file that I am going to use:
file1.txt:
School JUPAS_N Univ ID Working_line_number
TST 11-2 CUHK 90544
file2.csv:
Date,JUPAS,UniType,DSE,PRA ,PE,SIP,BRA,DDI Group,...
2
votes
3answers
39 views
Attach session numbers to its relevant IDs
I have long records were collected as an output of a code,
Part1=STS768976N-4BT345432-X00001-87675789AAAS Session=9
Part2=STS876578N-2BT234522-X00002-23456789UJHG Session=10
Part3=ST9876887N-...
8
votes
5answers
567 views
Replace the third comma on each line with newline
I have these long IDs which consist of four sections:
AKJHGFGUIKL,OIUYT,KJHBTYUI,98765434567
RTYUIKHGFGH,TYUJI,TGHYJKJKLJKL,6789876
ETRYTUUI,YTYUIL,UIOKJHGFGH,34567898766
I want to put the numbers ...
6
votes
2answers
548 views
How to pop (read&remove) a line of a file?
I am trying to get the last line of a file and then delete it:
Read the line:
sed -n '$p' file
# or
tail -n 1 file
Remove the line:
sed -i '$d' file
This works, but is there a way to do this with ...
0
votes
0answers
15 views
Sequentially print data output as a formatted table in Python [migrated]
I have written a Python script to execute data like
My script :
import os
import os.path
import re
import smtplib
from email.mime.text import MIMEText
infile = r"D:\i2Build\i2SchedulerReport.txt"
...
0
votes
1answer
30 views
Print file Word for Word on keypress
I want to do the following in bash: I have a textfile, and want to call a script that reads the file, waits for keystrokes from me and each time a correct button is pressed, it prints exactly one word ...
1
vote
2answers
30 views
How to match a column from File1 to get its corresponding fasta sequences in File 2?
May I kindly ask how to match all values of the first column in File with the line text in File 2 so I can copy the fasta sequences of all Object ID in File 1?
File 1.csv file
Object_ID, Length, ...
1
vote
3answers
35 views
If a line starts with {condition}, insert variable after 187th character
I have a .txt file with >10k lines. Some of the lines start with 1,POS,MGC=or 1,NEG,MGC=. For these lines, I want to insert a variable after the 187th character on the line while preserving the text ...
4
votes
3answers
207 views
Removing caret sign and adding parentheses to pager numbers
I have strings like these
NE234GJKLKU,*9^789098
NE345HJsdfe,*1^534656
YBKJNJKHBKK,*1^987654
UTGHNKOIUYO,*1^123421
ERTYUIJHGLK,*1^456666
that I wish to be:
NE234GJKLKU,*(978)-9098
NE345HJsdfe,*(153)...
0
votes
3answers
105 views
Field separator part of a column - incorrect parsing unix
I want to do a check the number of columns in a CSV file before processing it. The problem is that the delimiter (comma) also occurs in the text of some fields and because of that I cannot parse it ...
1
vote
2answers
25 views
link MAC adresses and port IDs
I have text file which include MAC address and its relevant ports, these were ordered in a way where the MAC address matches the single number of the same line numbers, in other words MAC address on ...
4
votes
3answers
47 views
extract paragraph match with words using sed or awk [duplicate]
I have file called "quota.txt" as below:
DISK_info_start:/usr
Disk: /usr
CURRENT=81
DISK_info_end:/usr
DISK_info_start:/usr/var
Disk: /usr/tmp
CURRENT=1
DISK_info_end:/usr/var
I have a variable as ...
0
votes
1answer
35 views
To compare column1 of two files
I need to compare the column 1 of file 1 with the column 1 of file2 in my script, if the column 1 of file1 matches with the column 1 of file2, then only it should proceed further ,else exit.
I'm with ...
2
votes
1answer
20 views
Adding missing colon to time column in a table
I have table like this one:
"Date","Time","Open","High","Low","Close","Volume"
06/28/2004,0931,37.49,37.50,37.45,37.46,1049200
06/28/2004,0932,37.48,37.50,37.45,37.48,450700
06/28/2004,0933,37.48,37....
10
votes
4answers
488 views
remove comma from a specific portion of a long string
I have document contain very long strings represent decoded information, i want to remove , commas from the last portion of the string. For example this is one of the long strings:
Insert into ...
4
votes
3answers
53 views
find IDs of databases with error during upgrade
I have a log file of errors captured in my database, the file contains a list of the records that had errors and were not updated. The record names are 10 digits and start with a strA:
Error: could ...
4
votes
3answers
183 views
Replacing just the first occurence of a character in a line
I have a line with thousands of lines like this:
{"100K";"0.00001";"10";"0.01"]}
{"101K";"0.0001";"100";"0.1"]}
{"102K";"1";"1000";"1"]}
{"102K";"1";"1000";"1"]}
{"103K";"0.01";"10000";"10"]}
{"104K";...
0
votes
2answers
298 views
Shrinking text in a file using bash [duplicate]
I have text file look like this:
If you are a software developer in your 20s or 30s, you've grown up in a world dominated by Linux. It has been a significant player in the data center for decades, ...
1
vote
1answer
25 views
What are accents in Xcompose?
I have been trying to figure out this thing called XCompose. Apparently if I use it correctly, I can do things like writing é or any such letter easily but don't understand how it works:
I did do the ...
