Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

352 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
4answers
167 views

Compare lines and upgrade two different files

I have a couple of text files with the following features: $ cat file_1 Line A Line B Line C Line D $ cat file_2 Line A Line added 1 Line B Line D Line added 2 They are such that file_1 has some ...
3
votes
0answers
54 views

Git - prune every whitespace-separated word originally introduced by specific author in project's history

We have a project under git revision control with only a single branch. We need to remove every new whitespace-separated word that was introduced for the first ever time in a given file by a specific ...
3
votes
2answers
59 views

Find the smallest set of values where at least one value is present in every row of a file

I have a file df like: 1 4 1 6 1 7 1 10 2 3 2 9 2 10 3 4 4 7 9 10 I need to identify the smallest set of values such that every row in df contains at least one of these values. ...
3
votes
3answers
111 views

search column 2 in csv file for value, if value, then insert “invalid” and shift cells right

I have csv file that is auto generated by a script but for some of the records (line items) received I need to search column 2 and if the values contains "*.app" I need to print "INVALID" into column ...
3
votes
1answer
100 views

Leftover characters after comparing two strings

I'm currently writing a game in bash that will compare user input against computer input. I want to find the leftover characters after a comparison between two strings. Below is what I have in mind: ...
3
votes
3answers
170 views

How to extract logs between two time stamps, line may or may not start with datetime

How to extract logs between two time stamps, line may or may not start with datetime. I am trying below and it is only extracting the lines that starts with datetime only. Datetime format is 2014-04-...
3
votes
1answer
630 views

Text editor that allows for coloring of arbitrary text (and exporting thereof)?

I have a rather large backtrace, which is kind of difficult to read, so I thought I'd color the portions of the backtrace text for easier reading. (I thought, as a backtrace problem, this question is ...
3
votes
1answer
1k views

Modify filenames in Makefile

For reasons that are not important here, I have a source code that I process automatically file-by-file and the processed source files are renamed in a systematic manner. For example, I start out with ...
2
votes
0answers
22 views

extract last 5 mins of information from the log file

system date format: Thu Jun 13 12:55:18 EDT 2019 My log format: 193.118.26.141 - - [01/May/2019:00:08:49 -0400] "GET / HTTP/1.1" 200 27 can someone please help me, how to get last 5 minutes of ...
2
votes
1answer
78 views

Remove deduplicates in a very large file()

In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4. so in the output there is only one row with a, one row with ...
2
votes
1answer
46 views

Script to add a block in a few hundred files

I want to add the following lines if ($http_user_agent ~* "somewebsite" ) { return 444; } to all my vhosts in NGINX as follows server { location / { want_to_add_those_3_lines_here } } ...
2
votes
0answers
33 views

Process (not remove) carriage return (^M) characters?

Is there a way to read in and process control characters like ^M and only show the final output? For example, I have the following string: track^Mfl^Ms I would like to convert it to: slack How ...
2
votes
2answers
26 views

edit a file using the values in another file as an input and redirect the file edit into another file

So basically I've a file that contains the below Data.txt <IP Address1>, 10, 23, <GW IP1> FINAL INPUT.45.324 <IP Address2>, 40, 33, <GW IP2> Another file that has values for ...
2
votes
2answers
83 views

merging two files according to the specified row

I have file1.txt looks as follows: search http://google.com mail https://gmail.com file2.txt looks like this: This is a search tool This is your mail account I need to merge them in the way ...
2
votes
2answers
60 views

How can I prepend lines that don't start with certain keywords?

I'm modifying a bunch of python files and I need to prepend every variable declaration with 'self.'. Here's an example of what I'm trying to do. From this: class testClass(parentClass): def ...
2
votes
1answer
54 views

Reading values from a file and printing those lines present in another file

I have two different files: file1 with 1 million entries and file 2 with 1.3 millions entries. My file 1 looks like this A 54 54 67 abcd C 54 4 1.2 lmno D 43 2 22.2 asdasd File2 looks similarly A ...
2
votes
2answers
400 views

Parsing FIX messages for further analytics

I'm currently trying to parse my FIX messages to get 2 columns showing currency (tag55) and price (tag133) but having difficulty using 'awk' as it appears the desired part of the message isn't split ...
2
votes
1answer
160 views

Any Text File Merge Tools?

When I do version control on text (or latex) files I often run into the problem that the merge/diff tools are all line based while text files are not. This leads to insufficient granularity: every ...
2
votes
1answer
240 views

extract fasta entries from list using while read

I have 28 files that each have ~14,000 "entries". A single entry consists of a header, denoted by >string, a newline, and then a sequence which is a string. Each entry has variable length sequence/...
2
votes
1answer
93 views

replacing values in one with the values in another file

I have a csv file named eche in following format: INCON,--,INITIAL,CONDITIONS,FOR*****,ELEMENTS,AT,TIME ,0.315570E+13 VC76,0.10000000E+00,0.2837726135782E+08,0.6756896308414E+02 1K02,0.10000000E+00,0....
2
votes
0answers
33 views

Timed display of text in terminal, perhaps slight markup (bold, colors). A.K.A. “play .srt subtitles”

Independent of the media player, I want to display separate subtitles in a scrolling text terminal to make language learning easier, or when I want more/different subtitles than the other viewers, ...
2
votes
2answers
464 views

Edit inotifywait output in daemon mode

I've been making progress, but I'm not sure how to proceed from here. I'm using inotifywait with the -d (daemon) switch to output a file that lists the video files that are put into the watched folder....
2
votes
2answers
706 views

Why the output of “column” is misaligned with a ANSI colored input?

I'm working on a commandline that retrieve some data (curl), extract the relevant fields (awk) and format it (column). It work nice, although it's very ugly (but all my script begin from a "too long ...
2
votes
2answers
234 views

How to copy strings from middle of lines (between strings) to the end of lines

I have a text file (1.txt) as follows: word1 word2 word3 I use sed to add strings to the beginning and end of every line as follows: sed 's/^/blahblah1 /g' 1.txt > 2.txt sed 's/$/ blahblah2/g' ...
2
votes
1answer
64 views

Join two files each with two columns that have multiple columns are matched

I have two files File A MUREX_2089975 :SBE Zinc Non-Option 118510 metals USD -308410 MUREX_2097300 :SBE Zinc Non-Option 118510 metals USD 7751 MUREX_2097474 :SBE Zinc Non-Option ...
2
votes
0answers
489 views

Tool to convert arbitrary MIME emails to plain text? (best effort)

I'm wondering if there is a command-line tool to do this: Take as input an arbitrary MIME email and output a best-effort guess at a plain-text version of the email message. Here is my ...
2
votes
2answers
555 views

How to format streaming JSON output from Node.js?

I have a Node.js server that uses JSON logging. When running the server locally, it outputs log entries as JSON objects which are quite tedious to read. I would like to pipe the output through a ...
2
votes
2answers
814 views

opposite of join -v?

I have to do just the opposite of join -v1 fileA fileB, which says: print lines from fileA whose first fields do not match the first field of any line from fileB. Instead, I want to write the lines ...
2
votes
1answer
58 views

Rejoin words breaked by hyphen and double newline characters

I have converted some PDF files to text by Calibre. Sample text as below: The booming population, coupled with a development strategy pre- mised on high levels of investment, meant that an ...
2
votes
0answers
342 views

convert attachments of type winmail.dat (TNEF) to “real” attachments in stored email messages, recursively in MailDir

Some email messages contain the attachments in a format called TNEF (Transport_Neutral_Encapsulation_Format). This attachment is usually called winmail.dat. Extraction tools exist for linux, but ...
2
votes
2answers
1k views

Read from and append to file at the same time while preserving end-of-line

Conceptually I have an easy task... I have [loosely] structured data in a file: Testing: debug, default CXXFLAGS <100's of additional output lines> Testing: release, default CXXFLAGS <100's ...
2
votes
0answers
860 views

how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
2
votes
1answer
56 views

Scraping from applications other than browsers on Debian Jessie

I'm thinking of building an automated trading system. I think there's probably no services supplying informations of order book and top of book in the txt format. So I have to do scraping. I ...
2
votes
2answers
2k views

Add Filename as first line of file in shell script

Hello and thanks in advance. I need to take a file, insert the filename as the first line of the file, then move to a different name. Here's the wrinkle. I need to grab the oldest file in the ...
2
votes
2answers
181 views

Exchanging One Column From File1 with files from File2

I have two files. The first file has 11 columns, for example: 1 2 3 4 5 6 7 8 9 10 11 The second has 10 columns and might look like this: 11 22 33 44 55 66 77 88 99 100 What I want to do is look ...
2
votes
3answers
1k views

Grep for multiple strings, show number of lines after one of the strings (but not the other)

I would like to search a file for multiple strings and for one particular string, show the 5 lines following it as well. cat file | grep -e 'string1' -e 'string2' works of course, but I would like ...
2
votes
1answer
178 views

Convert / import flat file with multiple records, one variable per line

I am searching for an efficient way using (UNIX) command line tools to read a flat file into a(ny) database, perhaps after first converting to an intermediate structured format, for example XML or csv....
2
votes
2answers
172 views

Using “ps -C java -o pid,user,cmd” Can I reduce the output of cmd?

I am trying to get this command down to just the essential information that I need. In the command cmd comes back with this output... /usr/local/jre1.7.0_51/bin/java -Dwd.tag=boxect -Dwd.instance=...
2
votes
0answers
260 views

Shell scripts for the database logs

The actual log file will be like this $ cat file1.log time=2014-07-23 23:56:28 GMT, user=[unknown], db=[unknown], host= pid=28254 LOG: time=2014-07-23 23:56:28 GMT, user=portalman, db=ss, host=172....
2
votes
2answers
771 views

Insert a line after a block of text with sed

I have text file: file.conf ### Option: LogFile # Name of log file. # If not set, syslog is used. # # Mandatory: no # Default: # LogFile= ### Option: LogFileSize # Maximum size of ...
1
vote
1answer
44 views

Find the different lines between two files without sorting

I have two files, file1 is a subset of file2, which means all the lines in file1 can be found in file2 but some lines in file2 are not in file1. Now I want to find the different lines(or extra lines) ...
1
vote
0answers
6 views

Train spamassassin remotely using a single email account set up for both POP and IMAP

I use multiple devices ( desktop / laptop / iphone / ipad ). I use POP because my desktop has extensive folders that I do not want to upload/share with a server, or take up space on my mobile devices. ...
1
vote
3answers
89 views

How to move lines containing specific strings to top of file?

How can I move lines containing specific strings (ie. 005, and 007) to the top of the target file? Contents of target file: XXX_008 XXX_001 XXX_002 XXX_009 XXX_003 XXX_006 XXX_005 ----> located ...
1
vote
1answer
80 views

Improve performance when using “system” call (shell escape) processing large files in awk

I have an awk script that processes very large files that look something like this: K1353 SF3987.7PD833391.4 KARE K1353 SF3987.2KD832231.4 MEAKE K1332 IF4987.7RP832231.2 LEAOS K1329 SF2787.7KD362619....
1
vote
2answers
51 views

replace header in a file with list of lines in another file

I have a fasta file contained ~28000 sequence. I want to replace header of these sequences by a list of lines in another file. Example: File 1: sp|B7UM99|TIR_ECO27 MPIGNLGNNVNGNHLIPPAPP..... sp|...
1
vote
1answer
32 views

Element comparison between subsequent lines, assigning a value for each null output

I have an output which looks like this: ASCII file I want to pick up the last element from the rows that contain the string "Fe\s2b". But the thing is each occurrence of "H\s\s1\s\s\s\s\s\s4861.33A" ...
1
vote
1answer
39 views

Calculate difference in numbers and save in colon-separated time format with bash script

I have a tab-separated file, e.g., 0018803 01 1710 2050 002571 0018951 01 1934 2525 003277 0019362 02 2404 2415 002829 0019392 01 2621 2820 001924 0019542 01 ...
1
vote
1answer
139 views

Save Request url in Curl output

The command below cats a file full of urls and feeds it into gnu parallel. I would like the output in curlresponse.txt to have also the original request url. How can it be done? cat urls.txt | ...
1
vote
3answers
56 views

Search for a dynamic pattern in a file in a file and replace it with variables

I have written the below commands that will generate three different shuffled vales A=`echo 'abcdefghijklmnopqrstuvwxyz' | sed 's/./&\n/g' | shuf | tr -d "\n"` B=`echo '...
1
vote
0answers
28 views

Merge 2 files based on the 2nd column of each and overwrite common lines in first file

I have a couple of CSVs'. file1 has a large number of URLs and file2 also has a URL field which have to be matched with file1. Now file1 has a couple of columns I need to add to file2. I have managed ...