Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

-1
votes
1answer
16 views

Joining 2 files and excluding word from third file

I have three files: file1, file2 and exl. Both file1 and file2 have two columns of content. The first column is the same in both files. I want to join the content of both files using that column. ...
1
vote
3answers
30 views

Remove all parent directories from a list of paths

I have a bash command that lists the directories in HDFS beginning with /random. It goes as follows: hdfs dfs -ls -R /random | grep '^d' | awk '{print $NF}' It generates the following output: /...
2
votes
1answer
32 views

`sed` updates the meta dates of all accessed files

I used sed -i 's/oldpat/newpat/g' ~/mydir/**/*.txt to replace some str in some text files. However, the modified dates of all files, including those with contents unchanged, were changed to current ...
0
votes
2answers
20 views

Copy values from a file to a new file only after certain characters

I have a file with the following format: Reading frame 3000 time 30000.000 ^MReading frame 3000 time 30000.000 mu 8.788e+00 <mu> 2.945e-03 and I want to paste the values which are ...
0
votes
1answer
33 views

cumulative table, create new table in which values in each column are sum of each previous column values [on hold]

I am dealing with script to transform table - in output table each column should be a sum of previous columns (something like cummulative table). Table got unknown number of rows and columns, first ...
1
vote
3answers
37 views

How to loop through first file and print each row in the first file next to all the rows in the second file using AWK?

I think the title may be confusing, so here's a small example of what I am trying to accomplish. File 1: Adam John Tim File 2: ,2018-2-1301 ,2018-3-1379 ,2018-1-3809 ,2018-2-5801 Output: Adam,...
1
vote
2answers
48 views

remove first two bytes of a line

I have a file that has 2 bytes of binary indicator values in every row that has to be stripped off. I tried sed, awk, cut but none of them work (though they work in vi mode): sed 's/^.\{,2\}//' file....
0
votes
2answers
56 views

Print the lines that only with all consecutive repeated characters

I have a text file and I want to print only the lines that have all their characters are consecutively reapeated at least two times or more. FOR EXAMPLE THE INPUT: 11 AAA 555227777 BBhh@@222 baabbb ...
0
votes
3answers
18 views

count matching open ports in a network log file

I have network log file like this one: Nmap scan report for 192.168.1.51 Host is up. PORT STATE SERVICE 80/tcp open http 443/tcp open https 8080/tcp open http-proxy 443/tcp ...
6
votes
5answers
734 views

Making summary of sentences

I have data and I want to summarize sentences to generate conclusions. The example below is not related to the data, but just to clarify the idea so I can replicate it. Employee Suzie signed one time....
1
vote
1answer
43 views

How to remove pattern from line but return what remains?

If I run ip addr sho | grep 9000 it returns the interface that I need quick stats for: eth3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000 However, I'd like to ...
1
vote
3answers
58 views

extract, cut, process strings in a bash script

I have basically 4 txt files with records such as FILE 1 localhost_access_2018-07-19.tar.gz:13 localhost_access_2018-07-20.tar.gz:17 localhost_access_2018-07-21.tar.gz:12 localhost_access_2018-07-...
0
votes
4answers
42 views

unix sed remove adapter names

I would like to pipe the output to sed and remove the below adapter names using sed and group the values with commas ent13: 16 3979 ent12: 211 411 3978 ent11: 15 3977 ent10: 111 178 179 311 511 1719 ...
4
votes
3answers
132 views

Deleting portion of a text file and following lines using sed

I need to edit a file like the following: auto wlx00 allow-hotplug wlx00 iface wlx000 inet dhcp iface wlx000 inet6 auto post-up sysctl -w net.ipv6.conf.wlx000.accept_ra=2 auto wlx000 the goal is ...
5
votes
4answers
528 views

How can I find all lines containing two specified words? [duplicate]

I need to check if two (specified) words exist on any line in a text file. There are no limits for the characters of the words. For example: I want to find lines of a text file that contain the two ...
0
votes
1answer
23 views

how to copy columns from file 2 to file 1 when some columns matches between 2 files

I have a file head top_candidates 25 elevation_e gene1 20 9 0.0246022994932004 5 8 10.9217937824527 30 elevation_e gene1 59 18 0.0246022994932004 7 12 15.653559774527 31 elevation_e ...
-4
votes
3answers
80 views

File Processing to transpose data [on hold]

I have a file with IDs and names applicable to them as below: 1234|abc|cde|fgh 5678|ijk|abc|lmn 9101|cde|fgh|klm 1213|klm|abc|cde I should get the output as different files having IDs matching each ...
0
votes
2answers
45 views

how can i move lines in mulitple text files? [duplicate]

Hi assume I have a files that look like: x x x A B C 1 2 3 D E F x x x and what need the files to look like: x x x A B C D E F 1 2 3 x x x I know I could do this with a few sed -i s/search/replace/...
17
votes
3answers
1k views

Removing all non-ascii characters from a workflow (file)

How would I remove all non-ascii characters from one file? Would there be a specific command to perform this? grep --colour='auto' -P -n'[^\x00-\x7]' /usr/local/... I believe this finds the ...
-2
votes
2answers
48 views

Find longest stretch of pattern from centre of sequence

How to calculate the longest consecutive pattern of C/T anywhere in sequence and longest consecutive pattern of C/T from centre CCCCTGTTGCCAAACAATGC TTTTCCCGCCTTTGGCCTAC ...
0
votes
1answer
28 views

Extract names from File_B having overlapping intervals with File_A

Two space-delimited files: File_A MT 50000 groupI 7850000 groupI 7950000 groupI 9050000 groupI 21750000 groupII 8750000 groupII 10550000 groupII 16150000 groupII 20850000 groupIII 14750000 groupIII ...
-1
votes
0answers
20 views

Regular expression to match only last folder [duplicate]

I would like to get the name of the last folder in a path for use in bash script. For example: I have a path /media/devicename/folder1/ I would like to get a variable called foldername that has in it ...
3
votes
3answers
546 views

Find how many times a certain DNA base sequence occurs in a file

The assignment is to write a bash script named “countmatches” that will display the number of times a certain sequence, such as aac, appears in a specified file. The script should expect at least two ...
1
vote
2answers
174 views

Add four lines between sentences [duplicate]

Is there an easy way to add four empty lines between each sentence. For example, this paragraph: This combination of vitamins and selected minerals is used to prevent or treat vitamin deficiency ...
4
votes
2answers
66 views

removing digits from the end of lines in text file

I have this sample from a log file which is driving me crazy. i want to remove the last 8 digits but the problem there are colons in different places so I tried awk but it didn't work. this is the ...
0
votes
3answers
65 views

How to split a file into paragraphs and name the resulting pieces based on an identifier present in each paragraph

I have a big file with more than 3264880 lines. I wanted to split that file based on Two strings "BEGIN JOB" and " END JOB" and write it into multiple files and the file name should be based on a ...
0
votes
2answers
80 views

Unix script to read the file and create a file for each line

I have a file and my task is to read the file and generate a new file for each line. The name of newly generated file should come from the line content. Example The file has these two lines: My ...
0
votes
1answer
14 views

Filter file to list all paths in it

How can I filter text file to remove every thing and just leave the paths in it? For example: dslkjcblasdjncklasdnlc /asdds/ads/adss/sdf.jpg asljdascuashn sdkm /sdf/dsf/dsf/dfs/sdfs.txt ,nbaskhdnasm....
0
votes
3answers
34 views

Replace space with Pipe between the tags

I have a file that is coming in tags "{}". Need to convert it into a pipe format. Here is the sample: {Id USXXXXXX} {Code USXXXXX} {Desc abcd 6.000% 5/24} Expected result 1: {Id|USXXXXXX} {Code|...
1
vote
4answers
55 views

How to move a line to the previous line if a certain character “ : ” is not found? [duplicate]

I have a lot of text files of transcripts. I have cleaned it to an extent. The last bit of cleaning is the following. I have this in certain files *.txt Gary: I said something. Larry: I said ...
4
votes
4answers
58 views

How to compare the 1st column of two files and if the strings match then print “true” [duplicate]

I have two files, say file1 and file2. file1: 91 | 2018-10-11 16:07:14 | bkwdv | 188.0000 | 77 | 2018-10-12 15:05:15 | wvwef | 94.0000 | file2: 77 | 2018-10-12 15:05:15 | wvwef | 94.0000 91 | ...
-2
votes
1answer
58 views

How to do awk to do sorting between a txt file and a csv file? [closed]

Here is the file that I am going to use: file1.txt: School JUPAS_N Univ ID Working_line_number TST 11-2 CUHK 90544 file2.csv: Date,JUPAS,UniType,DSE,PRA ,PE,SIP,BRA,DDI Group,...
2
votes
3answers
39 views

Attach session numbers to its relevant IDs

I have long records were collected as an output of a code, Part1=STS768976N-4BT345432-X00001-87675789AAAS Session=9 Part2=STS876578N-2BT234522-X00002-23456789UJHG Session=10 Part3=ST9876887N-...
8
votes
5answers
567 views

Replace the third comma on each line with newline

I have these long IDs which consist of four sections: AKJHGFGUIKL,OIUYT,KJHBTYUI,98765434567 RTYUIKHGFGH,TYUJI,TGHYJKJKLJKL,6789876 ETRYTUUI,YTYUIL,UIOKJHGFGH,34567898766 I want to put the numbers ...
6
votes
2answers
548 views

How to pop (read&remove) a line of a file?

I am trying to get the last line of a file and then delete it: Read the line: sed -n '$p' file # or tail -n 1 file Remove the line: sed -i '$d' file This works, but is there a way to do this with ...
0
votes
0answers
15 views

Sequentially print data output as a formatted table in Python [migrated]

I have written a Python script to execute data like My script : import os import os.path import re import smtplib from email.mime.text import MIMEText infile = r"D:\i2Build\i2SchedulerReport.txt" ...
0
votes
1answer
30 views

Print file Word for Word on keypress

I want to do the following in bash: I have a textfile, and want to call a script that reads the file, waits for keystrokes from me and each time a correct button is pressed, it prints exactly one word ...
1
vote
2answers
30 views

How to match a column from File1 to get its corresponding fasta sequences in File 2?

May I kindly ask how to match all values of the first column in File with the line text in File 2 so I can copy the fasta sequences of all Object ID in File 1? File 1.csv file Object_ID, Length, ...
1
vote
3answers
35 views

If a line starts with {condition}, insert variable after 187th character

I have a .txt file with >10k lines. Some of the lines start with 1,POS,MGC=or 1,NEG,MGC=. For these lines, I want to insert a variable after the 187th character on the line while preserving the text ...
4
votes
3answers
207 views

Removing caret sign and adding parentheses to pager numbers

I have strings like these NE234GJKLKU,*9^789098 NE345HJsdfe,*1^534656 YBKJNJKHBKK,*1^987654 UTGHNKOIUYO,*1^123421 ERTYUIJHGLK,*1^456666 that I wish to be: NE234GJKLKU,*(978)-9098 NE345HJsdfe,*(153)...
0
votes
3answers
105 views

Field separator part of a column - incorrect parsing unix

I want to do a check the number of columns in a CSV file before processing it. The problem is that the delimiter (comma) also occurs in the text of some fields and because of that I cannot parse it ...
1
vote
2answers
25 views

link MAC adresses and port IDs

I have text file which include MAC address and its relevant ports, these were ordered in a way where the MAC address matches the single number of the same line numbers, in other words MAC address on ...
4
votes
3answers
47 views

extract paragraph match with words using sed or awk [duplicate]

I have file called "quota.txt" as below: DISK_info_start:/usr Disk: /usr CURRENT=81 DISK_info_end:/usr DISK_info_start:/usr/var Disk: /usr/tmp CURRENT=1 DISK_info_end:/usr/var I have a variable as ...
0
votes
1answer
35 views

To compare column1 of two files

I need to compare the column 1 of file 1 with the column 1 of file2 in my script, if the column 1 of file1 matches with the column 1 of file2, then only it should proceed further ,else exit. I'm with ...
2
votes
1answer
20 views

Adding missing colon to time column in a table

I have table like this one: "Date","Time","Open","High","Low","Close","Volume" 06/28/2004,0931,37.49,37.50,37.45,37.46,1049200 06/28/2004,0932,37.48,37.50,37.45,37.48,450700 06/28/2004,0933,37.48,37....
10
votes
4answers
488 views

remove comma from a specific portion of a long string

I have document contain very long strings represent decoded information, i want to remove , commas from the last portion of the string. For example this is one of the long strings: Insert into ...
4
votes
3answers
53 views

find IDs of databases with error during upgrade

I have a log file of errors captured in my database, the file contains a list of the records that had errors and were not updated. The record names are 10 digits and start with a strA: Error: could ...
4
votes
3answers
183 views

Replacing just the first occurence of a character in a line

I have a line with thousands of lines like this: {"100K";"0.00001";"10";"0.01"]} {"101K";"0.0001";"100";"0.1"]} {"102K";"1";"1000";"1"]} {"102K";"1";"1000";"1"]} {"103K";"0.01";"10000";"10"]} {"104K";...
0
votes
2answers
298 views

Shrinking text in a file using bash [duplicate]

I have text file look like this: If you are a software developer in your 20s or 30s, you've grown up in a world dominated by Linux. It has been a significant player in the data center for decades, ...
1
vote
1answer
25 views

What are accents in Xcompose?

I have been trying to figure out this thing called XCompose. Apparently if I use it correctly, I can do things like writing é or any such letter easily but don't understand how it works: I did do the ...