Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

410 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
4answers
224 views

Compare lines and upgrade two different files

I have a couple of text files with the following features: $ cat file_1 Line A Line B Line C Line D $ cat file_2 Line A Line added 1 Line B Line D Line added 2 They are such that file_1 has some ...
3
votes
3answers
48 views

regex from two files

I have two files that built of blocks of 4 lines, starting with a @. They are matching and in order. The files are too big (>10 GB) so I am putting here the first 3 blocks of each (12 lines). 1st: ...
3
votes
2answers
159 views

Parsing csv file to filter rows basing on a matching set of characters from a column value

Consider the following csv file: A,3300 B,8440 B,8443 B,8444 C,304 C,404 M,5502 M,5511 The real csv file is really big (around 60,000 rows) but I only included a small version for ...
3
votes
0answers
58 views

Git - prune every whitespace-separated word originally introduced by specific author in project's history

We have a project under git revision control with only a single branch. We need to remove every new whitespace-separated word that was introduced for the first ever time in a given file by a specific ...
3
votes
2answers
63 views

Find the smallest set of values where at least one value is present in every row of a file

I have a file df like: 1 4 1 6 1 7 1 10 2 3 2 9 2 10 3 4 4 7 9 10 I need to identify the smallest set of values such that every row in df contains at least one of these values. ...
3
votes
3answers
305 views

search column 2 in csv file for value, if value, then insert “invalid” and shift cells right

I have csv file that is auto generated by a script but for some of the records (line items) received I need to search column 2 and if the values contains "*.app" I need to print "INVALID" into column ...
3
votes
2answers
890 views

Why the output of “column” is misaligned with a ANSI colored input?

I'm working on a commandline that retrieve some data (curl), extract the relevant fields (awk) and format it (column). It work nice, although it's very ugly (but all my script begin from a "too long ...
3
votes
1answer
106 views

Leftover characters after comparing two strings

I'm currently writing a game in bash that will compare user input against computer input. I want to find the leftover characters after a comparison between two strings. Below is what I have in mind: ...
3
votes
3answers
195 views

How to extract logs between two time stamps, line may or may not start with datetime

How to extract logs between two time stamps, line may or may not start with datetime. I am trying below and it is only extracting the lines that starts with datetime only. Datetime format is 2014-04-...
3
votes
1answer
856 views

Text editor that allows for coloring of arbitrary text (and exporting thereof)?

I have a rather large backtrace, which is kind of difficult to read, so I thought I'd color the portions of the backtrace text for easier reading. (I thought, as a backtrace problem, this question is ...
3
votes
1answer
2k views

Modify filenames in Makefile

For reasons that are not important here, I have a source code that I process automatically file-by-file and the processed source files are renamed in a systematic manner. For example, I start out with ...
2
votes
3answers
51 views

using filename and path as strings to be replaced using sed command

I am trying to deal with the following situation. I have many files (more than 100) with an .html extension in a folder tree and inside those files I have a string which is the file name. For example ...
2
votes
1answer
59 views

compare columns from two different files and PRINT RECORDS FROM FIRST FILE those that DO NOT match from second file

I would like to compare columns from file one to two. Where column 2 of file1 should un-match from column 1 or 2 from file2 and print output from file 1. file1. cat test.head20.R2.fastq.tab @...
2
votes
0answers
50 views

extract last 5 mins of information from the log file

system date format: Thu Jun 13 12:55:18 EDT 2019 My log format: 193.118.26.141 - - [01/May/2019:00:08:49 -0400] "GET / HTTP/1.1" 200 27 can someone please help me, how to get last 5 minutes of ...
2
votes
1answer
82 views

Remove deduplicates in a very large file()

In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4. so in the output there is only one row with a, one row with ...
2
votes
1answer
48 views

Script to add a block in a few hundred files

I want to add the following lines if ($http_user_agent ~* "somewebsite" ) { return 444; } to all my vhosts in NGINX as follows server { location / { want_to_add_those_3_lines_here } } ...
2
votes
0answers
34 views

Process (not remove) carriage return (^M) characters?

Is there a way to read in and process control characters like ^M and only show the final output? For example, I have the following string: track^Mfl^Ms I would like to convert it to: slack How ...
2
votes
2answers
30 views

edit a file using the values in another file as an input and redirect the file edit into another file

So basically I've a file that contains the below Data.txt <IP Address1>, 10, 23, <GW IP1> FINAL INPUT.45.324 <IP Address2>, 40, 33, <GW IP2> Another file that has values for ...
2
votes
2answers
85 views

merging two files according to the specified row

I have file1.txt looks as follows: search http://google.com mail https://gmail.com file2.txt looks like this: This is a search tool This is your mail account I need to merge them in the way ...
2
votes
2answers
67 views

How can I prepend lines that don't start with certain keywords?

I'm modifying a bunch of python files and I need to prepend every variable declaration with 'self.'. Here's an example of what I'm trying to do. From this: class testClass(parentClass): def ...
2
votes
1answer
62 views

Reading values from a file and printing those lines present in another file

I have two different files: file1 with 1 million entries and file 2 with 1.3 millions entries. My file 1 looks like this A 54 54 67 abcd C 54 4 1.2 lmno D 43 2 22.2 asdasd File2 looks similarly A ...
2
votes
2answers
547 views

Parsing FIX messages for further analytics

I'm currently trying to parse my FIX messages to get 2 columns showing currency (tag55) and price (tag133) but having difficulty using 'awk' as it appears the desired part of the message isn't split ...
2
votes
1answer
452 views

extract fasta entries from list using while read

I have 28 files that each have ~14,000 "entries". A single entry consists of a header, denoted by >string, a newline, and then a sequence which is a string. Each entry has variable length sequence/...
2
votes
1answer
107 views

replacing values in one with the values in another file

I have a csv file named eche in following format: INCON,--,INITIAL,CONDITIONS,FOR*****,ELEMENTS,AT,TIME ,0.315570E+13 VC76,0.10000000E+00,0.2837726135782E+08,0.6756896308414E+02 1K02,0.10000000E+00,0....
2
votes
0answers
43 views

Timed display of text in terminal, perhaps slight markup (bold, colors). A.K.A. “play .srt subtitles”

Independent of the media player, I want to display separate subtitles in a scrolling text terminal to make language learning easier, or when I want more/different subtitles than the other viewers, ...
2
votes
2answers
361 views

How to copy strings from middle of lines (between strings) to the end of lines

I have a text file (1.txt) as follows: word1 word2 word3 I use sed to add strings to the beginning and end of every line as follows: sed 's/^/blahblah1 /g' 1.txt > 2.txt sed 's/$/ blahblah2/g' ...
2
votes
1answer
94 views

Join two files each with two columns that have multiple columns are matched

I have two files File A MUREX_2089975 :SBE Zinc Non-Option 118510 metals USD -308410 MUREX_2097300 :SBE Zinc Non-Option 118510 metals USD 7751 MUREX_2097474 :SBE Zinc Non-Option ...
2
votes
0answers
530 views

Tool to convert arbitrary MIME emails to plain text? (best effort)

I'm wondering if there is a command-line tool to do this: Take as input an arbitrary MIME email and output a best-effort guess at a plain-text version of the email message. Here is my ...
2
votes
2answers
757 views

How to format streaming JSON output from Node.js?

I have a Node.js server that uses JSON logging. When running the server locally, it outputs log entries as JSON objects which are quite tedious to read. I would like to pipe the output through a ...
2
votes
1answer
65 views

Rejoin words breaked by hyphen and double newline characters

I have converted some PDF files to text by Calibre. Sample text as below: The booming population, coupled with a development strategy pre- mised on high levels of investment, meant that an ...
2
votes
0answers
398 views

convert attachments of type winmail.dat (TNEF) to “real” attachments in stored email messages, recursively in MailDir

Some email messages contain the attachments in a format called TNEF (Transport_Neutral_Encapsulation_Format). This attachment is usually called winmail.dat. Extraction tools exist for linux, but ...
2
votes
2answers
1k views

Read from and append to file at the same time while preserving end-of-line

Conceptually I have an easy task... I have [loosely] structured data in a file: Testing: debug, default CXXFLAGS <100's of additional output lines> Testing: release, default CXXFLAGS <100's ...
2
votes
0answers
975 views

how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
2
votes
1answer
57 views

Scraping from applications other than browsers on Debian Jessie

I'm thinking of building an automated trading system. I think there's probably no services supplying informations of order book and top of book in the txt format. So I have to do scraping. I ...
2
votes
2answers
197 views

Exchanging One Column From File1 with files from File2

I have two files. The first file has 11 columns, for example: 1 2 3 4 5 6 7 8 9 10 11 The second has 10 columns and might look like this: 11 22 33 44 55 66 77 88 99 100 What I want to do is look ...
2
votes
3answers
699 views

Using awk to merge 2 files

I would like to merge 2 files based on column 1, 2, 3. I have tried the awk command below but is not working. awk 'NR==FNR {h[$1FS$2FS$3]=$4; next}{k=$1FS$2FS$3; if (k in h) print $1,$2,$3,$4,h[k] ;...
2
votes
1answer
194 views

Convert / import flat file with multiple records, one variable per line

I am searching for an efficient way using (UNIX) command line tools to read a flat file into a(ny) database, perhaps after first converting to an intermediate structured format, for example XML or csv....
2
votes
2answers
180 views

Using “ps -C java -o pid,user,cmd” Can I reduce the output of cmd?

I am trying to get this command down to just the essential information that I need. In the command cmd comes back with this output... /usr/local/jre1.7.0_51/bin/java -Dwd.tag=boxect -Dwd.instance=...
2
votes
0answers
279 views

Shell scripts for the database logs

The actual log file will be like this $ cat file1.log time=2014-07-23 23:56:28 GMT, user=[unknown], db=[unknown], host= pid=28254 LOG: time=2014-07-23 23:56:28 GMT, user=portalman, db=ss, host=172....
2
votes
2answers
884 views

Insert a line after a block of text with sed

I have text file: file.conf ### Option: LogFile # Name of log file. # If not set, syslog is used. # # Mandatory: no # Default: # LogFile= ### Option: LogFileSize # Maximum size of ...
1
vote
1answer
24 views

egrep, sed or awk to get the number of active dhcp leases?

Let's say I have the following dhcpd.leases file: # The format of this file is documented in the dhcpd.leases(5) manual page. # This lease file was written by isc-dhcp-4.3.6-P1 # authoring-byte-...
1
vote
3answers
72 views

Combine columns in one file by matching header

I'm looking to take one file that has multiple lines and many columns, add (sum) the values horizontally of columns with headers that match. In my file, (not this example) I'm ignoring the first 9 ...
1
vote
3answers
88 views

How to replace a line in text using RegEX?

The problem I have is that is trying to match both sets of delimiter (above and below) I'm trying to match only the second part of the delimiter below (bolded). This is so I can add a new version ...
1
vote
1answer
28 views

Compare name of files to a column in a table and if a correspondence is found rename it according to another column

I have an extensive set of files in a directory (in bash) that are named (inconveniently) as follows: COMMONSTUFF_X_Y_**VARIABLE-IDENTIFIER**_COMMONSTUFF I also have a table in which one column ...
1
vote
1answer
47 views

How to add 'uniq -c' results and continuously overwrite in a loop?

I'm writing a shell script and I'd like to combine the results of the unique values and their counts one file at a time. For example, if I am reading the data from the the two files below (with the ...
1
vote
0answers
30 views

If I know the corresponding encoded bytes to some characters, how can I determine matching charsets?

I have a few files where the filenames obviously use the wrong charset. As soon as I know which charset is being used, I can use convmv to convert it to UTF-8. From the rest of the filename, I can ...
1
vote
1answer
73 views

Export command output specific lines and put it into CSV file

Could you assist me how to retrieve specific lines and put it into CSV file? How to process the output via Bash and Python3? Remark: "objectID" represents each found server "An instance of Path" ...
1
vote
1answer
93 views

How to find max value from specific XML element using a Perl one liner?

I have an XML file with the following syntax <element> <id>0</id> <tag1>something</tag1> ... <tagn>something</tagn> </element> <element> ...
1
vote
1answer
45 views

Compare first column in two files, if match: update date variable, else: append line to second file

So I have two files that look like this: file1 userName | cpu% | command | date created user1 101.6 plasma-de+ Thu Aug 8 09:30:17 MDT 2019 user2 100.0 plasma-de+ Thu Aug 8 09:30:17 MDT ...
1
vote
2answers
51 views

Merge two files searching by key and add at the end

I would merge data from two files into one adding only interesting Values from column. Searching key is UID in column 3 File #1. Also it should remove/omit UID duplicates and empty lines if appear. ...

1
2 3 4 5
9