Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
511
questions with no upvoted or accepted answers
6
votes
2
answers
189
views
What ever became of the "prep" program? And what can I use instead on modern Linux systems?
Long ago, in Seventh Edition UNIX (a/k/a V7), there was a program called prep. Its primary use was to take files of text, and break them up into one word per line, for further processing by other ...
- 1,166
4
votes
1
answer
321
views
Text file with unknown header does not print propperly
I have a bunch of text-files that always come along with a backup file:
file.TXT
file.BAK
the *.TXT file contains a header like this:
.V3 PL=62 MB=1 MT=1 DR=TMTYM FL=0 FT=0 ...
- 5,510
4
votes
3
answers
626
views
Parsing csv file to filter rows basing on a matching set of characters from a column value
Consider the following csv file:
A,3300
B,8440
B,8443
B,8444
C,304
C,404
M,5502
M,5511
The real csv file is really big (around 60,000 rows) but I only included a small version for ...
- 41
4
votes
1
answer
162
views
Leftover characters after comparing two strings
I'm currently writing a game in bash that will compare user input against computer input.
I want to find the leftover characters after a comparison between two strings. Below is what I have in mind:
...
- 65
4
votes
4
answers
792
views
Compare lines and upgrade two different files
I have a couple of text files with the following features:
$ cat file_1
Line A
Line B
Line C
Line D
$ cat file_2
Line A
Line added 1
Line B
Line D
Line added 2
They are such that file_1 has some ...
- 4,861
3
votes
1
answer
64
views
How can I pipe text that contains ANSI movement characters ESC[xC and ESC[xD and output the text formatted accodingly
I have text files that include the ANSI movement codes ESC[xC and ESC[xD. I want to filter these out but with each line output with those codes taken into account.
Consider the line:
this cat is ...
- 33
3
votes
1
answer
2k
views
Filter one very large CSV based on values from another CSV
I am processing some CSV files that do not fit in RAM.
The 2 CSV files have the following structure:
first.csv
id
name
timestamp
serial
str
yyyy-mm-dd hh:mm:ss
second.csv
id
name
date
serial
str
...
3
votes
2
answers
95
views
comm for n files
I am looking for comm's functionality for n, i. e. more than two, files.
man comm reads:
COMM(1)
NAME
comm - compare two sorted files line by line
SYNOPSIS
comm [OPTION]... FILE1 FILE2
...
- 31
3
votes
3
answers
193
views
regex from two files
I have two files that built of blocks of 4 lines, starting with a @. They are matching and in order. The files are too big (>10 GB) so I am putting here the first 3 blocks of each (12 lines).
1st: ...
3
votes
0
answers
81
views
Git - prune every whitespace-separated word originally introduced by specific author in project's history
We have a project under git revision control with only a single branch. We need to remove every new whitespace-separated word that was introduced for the first ever time in a given file by a specific ...
3
votes
2
answers
88
views
Find the smallest set of values where at least one value is present in every row of a file
I have a file df like:
1 4
1 6
1 7
1 10
2 3
2 9
2 10
3 4
4 7
9 10
I need to identify the smallest set of values such that every row in df contains at least one of these values.
...
- 31
3
votes
3
answers
1k
views
search column 2 in csv file for value, if value, then insert "invalid" and shift cells right
I have csv file that is auto generated by a script but for some of the records (line items) received I need to search column 2 and if the values contains "*.app" I need to print "INVALID" into column ...
- 171
3
votes
3
answers
336
views
How to extract logs between two time stamps, line may or may not start with datetime
How to extract logs between two time stamps, line may or may not start with datetime.
I am trying below and it is only extracting the lines that starts with datetime only. Datetime format is 2014-04-...
- 31
3
votes
0
answers
748
views
Tool to convert arbitrary MIME emails to plain text? (best effort)
I'm wondering if there is a command-line tool to do this:
Take as input an arbitrary MIME email and output a best-effort guess at a plain-text version of the email message.
Here is my ...
- 103
3
votes
0
answers
2k
views
how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?
I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
- 193
2
votes
1
answer
256
views
Text prediction in Linux while typing like on Android, iOS and Windows
I have noticed the utility of this feature while typing on Android devices (notably with Gboard virtual keyboard)
The same is available on iOS, in iPhones and iPads.
I don't mean the use of a virtual ...
- 1,398
2
votes
1
answer
305
views
AWK: Extract lines with values that match and less than values in another file?
I have a file called file1 on my Linux machine with these contents:
ID Time Energy
43 4.38665978376386365240533e-05 3.215e-02
43 4.38666558838421885677290e-05 4.788e-01
48 4....
2
votes
2
answers
523
views
using filename and path as strings to be replaced using sed command
I am trying to deal with the following situation. I have many files (more than 100) with an .html extension in a folder tree and inside those files I have a string which is the file name. For example ...
- 21
2
votes
1
answer
92
views
Why awk script is shuffling the last two fields?
My working file looks like this
Time[s] A_in B_in C_in D_in E_in
0.000000E+00 1.120200E-02 1.432000E-01 2.331500E-04 2.349800E-03 2.800000E-03
2....
- 33
2
votes
0
answers
76
views
Train spamassassin remotely using a single email account set up for both POP and IMAP
I use multiple devices ( desktop / laptop / iphone / ipad ). I use POP because my desktop has extensive folders that I do not want to upload/share with a server, or take up space on my mobile devices.
...
- 21
2
votes
2
answers
129
views
Remove duplicates in a very large file()
In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4.
so in the output there is only one row with a, one row with ...
2
votes
1
answer
62
views
Script to add a block in a few hundred files
I want to add the following lines
if ($http_user_agent ~* "somewebsite" ) {
return 444;
}
to all my vhosts in NGINX as follows
server {
location / {
want_to_add_those_3_lines_here
}
}
...
- 21
2
votes
0
answers
79
views
How can I regex match-and-replace _all_ on-screen text?
I need a way to replace an on-screen pattern with another string, regardless of what places that text on the screen. Maybe I can live without user input being replaced, but anything that comes from ...
2
votes
2
answers
128
views
edit a file using the values in another file as an input and redirect the file edit into another file
So basically I've a file that contains the below Data.txt
<IP Address1>, 10, 23, <GW IP1>
FINAL INPUT.45.324
<IP Address2>, 40, 33, <GW IP2>
Another file that has values for ...
- 21
2
votes
2
answers
107
views
How can I prepend lines that don't start with certain keywords?
I'm modifying a bunch of python files and I need to prepend every variable declaration with 'self.'. Here's an example of what I'm trying to do.
From this:
class testClass(parentClass):
def ...
- 21
2
votes
1
answer
107
views
Reading values from a file and printing those lines present in another file
I have two different files: file1 with 1 million entries and file 2 with 1.3 millions entries. My file 1 looks like this
A 54 54 67 abcd
C 54 4 1.2 lmno
D 43 2 22.2 asdasd
File2 looks similarly
A ...
- 141
2
votes
2
answers
1k
views
Parsing FIX messages for further analytics
I'm currently trying to parse my FIX messages to get 2 columns showing currency (tag55) and price (tag133) but having difficulty using 'awk' as it appears the desired part of the message isn't split ...
- 21
2
votes
1
answer
1k
views
extract fasta entries from list using while read
I have 28 files that each have ~14,000 "entries". A single entry consists of a header, denoted by >string, a newline, and then a sequence which is a string. Each entry has variable length sequence/...
- 21
2
votes
1
answer
249
views
replacing values in one with the values in another file
I have a csv file named eche in following format:
INCON,--,INITIAL,CONDITIONS,FOR*****,ELEMENTS,AT,TIME ,0.315570E+13
VC76,0.10000000E+00,0.2837726135782E+08,0.6756896308414E+02
1K02,0.10000000E+00,0....
- 21
2
votes
0
answers
56
views
Timed display of text in terminal, perhaps slight markup (bold, colors). A.K.A. "play .srt subtitles"
Independent of the media player, I want to display separate subtitles in a scrolling text terminal to make language learning easier, or when I want more/different subtitles than the other viewers, ...
- 5,906
2
votes
1
answer
677
views
How to monitor packets from a MAC address
I am wanting to monitor a MAC address for activity on my network using airodump-ng
I am currently:
First running this:
airodump-ng mon0 --write t
which writes to the file t-01.csv a csv with the ...
- 275
2
votes
2
answers
2k
views
How to copy strings from middle of lines (between strings) to the end of lines
I have a text file (1.txt) as follows:
word1
word2
word3
I use sed to add strings to the beginning and end of every line as follows:
sed 's/^/blahblah1 /g' 1.txt > 2.txt
sed 's/$/ blahblah2/g' ...
- 809
2
votes
1
answer
149
views
Join two files each with two columns that have multiple columns are matched
I have two files
File A
MUREX_2089975 :SBE Zinc Non-Option 118510 metals USD -308410
MUREX_2097300 :SBE Zinc Non-Option 118510 metals USD 7751
MUREX_2097474 :SBE Zinc Non-Option ...
- 21
2
votes
1
answer
5k
views
Need to find a response time which takes from 1-3 seconds in Apache logs
I need to find the API response time from an Apache log file. It's like a response time which is takes between 1 to 2 secound or 2 to 3 second. $6 is response time and values comes in microseconds.
I ...
- 23
2
votes
1
answer
78
views
Rejoin words breaked by hyphen and double newline characters
I have converted some PDF files to text by Calibre. Sample text as below:
The booming population, coupled with a development strategy pre-
mised on high levels of investment, meant that an ...
- 21
2
votes
0
answers
552
views
convert attachments of type winmail.dat (TNEF) to "real" attachments in stored email messages, recursively in MailDir
Some email messages contain the attachments in a format called TNEF (Transport_Neutral_Encapsulation_Format). This attachment is usually called winmail.dat. Extraction tools exist for linux, but ...
- 5,906
2
votes
1
answer
61
views
Scraping from applications other than browsers on Debian Jessie
I'm thinking of building an automated trading system.
I think there's probably no services supplying informations of order book and top of book in the txt format.
So I have to do scraping.
I ...
user144127
2
votes
2
answers
244
views
Exchanging One Column From File1 with files from File2
I have two files. The first file has 11 columns, for example:
1 2 3 4 5 6 7 8 9 10 11
The second has 10 columns and might look like this:
11 22 33 44 55 66 77 88 99 100
What I want to do is look ...
- 33
2
votes
3
answers
1k
views
Using awk to merge 2 files
I would like to merge 2 files based on column 1, 2, 3. I have tried the awk command below but is not working.
awk 'NR==FNR {h[$1FS$2FS$3]=$4; next}{k=$1FS$2FS$3; if (k in h) print $1,$2,$3,$4,h[k] ;...
- 21
2
votes
1
answer
322
views
Convert / import flat file with multiple records, one variable per line
I am searching for an efficient way using (UNIX) command line tools to read a flat file into a(ny) database, perhaps after first converting to an intermediate structured format, for example XML or csv....
user124838
2
votes
2
answers
205
views
Using "ps -C java -o pid,user,cmd" Can I reduce the output of cmd?
I am trying to get this command down to just the essential information that I need.
In the command cmd comes back with this output...
/usr/local/jre1.7.0_51/bin/java -Dwd.tag=boxect -Dwd.instance=...
- 21
2
votes
0
answers
393
views
Shell scripts for the database logs
The actual log file will be like this
$ cat file1.log
time=2014-07-23 23:56:28 GMT, user=[unknown], db=[unknown], host= pid=28254 LOG:
time=2014-07-23 23:56:28 GMT, user=portalman, db=ss, host=172....
- 57
2
votes
2
answers
1k
views
Insert a line after a block of text with sed
I have text file: file.conf
### Option: LogFile
# Name of log file.
# If not set, syslog is used.
#
# Mandatory: no
# Default:
# LogFile=
### Option: LogFileSize
# Maximum size of ...
- 21
1
vote
4
answers
109
views
Find text files where the content is duplicated?
By mistake I appended (>>) rather than overwrote (>) the output from a script that processed thousands of files, but I only did it for about 20 % of the files (I ran 5 parallel instances of ...
- 1,853
1
vote
0
answers
75
views
host hangs after cat /dev/null > bigfile.log
I found a big log file(2.7 TB) on my disk, so I decided to empty it with the following command:
cat /dev/null > bigfile.log
After I executed this cmd, I lost my ssh connection. When I logged in ...
- 11
1
vote
0
answers
160
views
Formatting multiline comments in source code (also for documentation)?
What I'm looking for can be stated simply as:
Are there any existing tools/approaches, that would allow me to (at least partially) automatically format C source code comments for documentation?
I'm ...
- 480
1
vote
1
answer
315
views
How to manipulate and extract certain data from a table?
I am quite new to Linux.
I have data like the following, it is in a tbl format
>Feature gnl|XXX|IFEJKLFI_79
locus_tag IFEJKLFI_05549
locus_tag IFEJKLFI_05549
...
- 11
1
vote
1
answer
336
views
Substitution in html file using awk
My awk script reads records from file1, finds the same records in file2 and substitutes alternate positions (of the record) with a defined symbol in that. But few of the values are not getting ...
- 27
1
vote
0
answers
182
views
How can I recursively search file contents by ANDed patterns and print the output like ag/the silver searcher would?
This will be easier to explain via example. Here are my input files:
file1:
x
x
a
b
c
x
x
file2:
x
x
c
b
a
x
x
file3:
x
x
x x a b c x x
x
x
file4:
x
x
x x c b a x x
x
x
file5:
x
x
a b
x
x
file6:
...
- 821
1
vote
0
answers
139
views
Invalid characters replace using VIM
In my file due to some reasons some invalid characters are coming making it very huge file. Sample below of one line. I am not sure where these characters are coming from(I am using google drive ...
