1

The problem

I have a 32M lines file with the following format

token^Iname^Iurl$

where ^I is the tab escape sequence, and $ is the end-of-line.

I need to get the url corresponding to not more than 10k matches with the field name. What I've done is

# Get second column
cut -f2 <myFile> |
# Find the word and line number
grep -nwi "<matchWord>" |
# Get just the number
cut -f1 -d ':' |
# Not more than 10k
head -n10000

And then, for each entry of the previous output

# Print line number 
sed -n '<number>{p;q}' <myFile>
# Get 3rd field
cut -f3

Now, this last operation with sed is ridiculously slow. I am wondering how to get the all of this by using grep only, or any other way that doesn't slow down after the first 1k matches.

Idea

It would be just perfect to be able to operate grep on the whole line (without cut -f2), targeting only the second column, and then cut -f3, but I don't have a clue of how to do it.

Example

Line xyz

qwertyuiop^Ibananas are yellow^Ihttp://mignons.cool$

Match word yellow in field name -> give me http://mignons.cool.

cut is needed, because I don't want to match stuff in the field token and url.

If I send to grep a cut of myFile, then I no longer have access to the url field, which I am interested in.

Input and expected output

Input file:

mxp4EdOy-IXkuwsuOfs0EQ^Ilegal yellow pad paper^I0/3/3031.jpg$
AeS7tgmlVffBhousr9YY5Q^Ihelicopter parking only sign^I0/3/3032.jpg$
8dl-VixSjG4Y0FpX9f5KHA^Iwritten list ^I0/3/3033.jpg$
XYvKZC3D_JSwlY8SPl-zLQ^Ihelicopter parking only road sign^I0/3/3034.jpg$
xF6zpvpHcmfpHP2MmT2FVg^Irun menu windows programming^I0/3/3035.jpg$
mCJvV2rXOmItLBkMZlyIwQ^Icoffee mug^I0/3/3040.jpg$
ZiobHk_dLsN-Q921KPJUTA^Icarpet^I0/3/3197.jpg$
xFrbGOMfVMl0WeqVAcT27A^Iwater jugs^I0/3/3199.jpg$

where ^I is the tab escape sequence, and $ is the end-of-line.

Match word helicopter.

Expected output (not more than 10k lines):

0/3/3032.jpg
0/3/3034.jpg

Potential solution

Since the url field contains only numbers, I could

cut -f 2,3 <myFile> | grep <matchWord> | cut -f2 | head -n10000

But it would be nicer to grep the second field only...

  • I don't quite get what you are trying to do but I am pretty sure awk can do a better job, replacing your usage of cut and grep. – phk Oct 26 '15 at 2:13
  • I put an example at the end, @phk. – Atcold Oct 26 '15 at 2:16
  • Took some time to figure out that ^I is <TAB>. Nice job… – Arthur2e5 Oct 26 '15 at 2:21
  • @Arthur2e5, I thought it was implicit (if you use cat -A, that's how they are marked). Anyhow, I've updated the question for ease of understanding. Tnx for the comment. – Atcold Oct 26 '15 at 2:24
  • Give as an example a few lines from the input file and desired output. – jimmij Oct 26 '15 at 2:35
1

You probably should not try to cut cut out. In fact, trying to consolidate a pipeline into a single process for the handling of 32M input lines will very likely negatively affect your task's overall completion time. This depends, though, on the kind of computer on which you run the job.

If the machine on which you process your data has multiple processor cores, then, generally speaking, consolidating a task loop to a single process means consolidating the whole job to a single processor core. This may be desirable on systems with only a single processor core, or else if overall CPU time is precious, but, in my experience, it is better to saturate a processor and use all cores concurrently to complete the task sooner.

That said, you definitely can grep just the second field:

grep -E $'\t(.* )?yellow( .*)?\t' <infile

...that pattern will match only strings which occur between two tab characters on a line, and will match only those strings which are bounded on both sides with either a space or one of the field delimiting tabs. With GNU grep you can also add the -max match switch for limiting output to no more than 10K matches. And so...

grep -m10000 -E $'\t(.* )?yellow( .*)?\t' <infile | cut -f3

...would be enough to do the whole job.

  • Sweet! I'm already saturating all cores by matching 5k different <matchWord> using multiple threads. What about the syntax you've used? Could you explain it a little more, perhaps? The use of parenthesis and ? are a tad obscure. – Atcold Oct 26 '15 at 6:08
  • @Atcold - do you have a specific question about it? its extended regexp is all, really. – mikeserv Oct 26 '15 at 6:10
  • The use of parenthesis and ? are a tad obscure. Moreover, once I have the matches, I have to move files around (~500GB, overall), so the bottleneck becomes the disk access time... – Atcold Oct 26 '15 at 6:15
  • 1
    @mikeserv: Sure, which is why I didn't post it as an answer, however looking at the sample data, it does not look at all that this be the case. Then again, the difference in speed versus accuracy does not warrant my proposal to even be considered seriously. Your accurate solution takes 2.7s for the same test with 100000 matches and 280ms for 10000 matches. – Moreaki Oct 26 '15 at 8:05
  • 1
    @Atcold - you have to match the tabs. if you dont put literal tab characters in your pattern, then youve got to get them in there somehow otherwise. thats why i use $'...' here because its hard to screw up. but if you dont want to use it then youll need actual tabs chracters rather than the backslash esacape sequence \t. – mikeserv Oct 28 '15 at 3:33
2

There are many ways to do this. Easiest is probably with awk

$ awk -F$'\t' '$2 = /helicopter/ {print $3}' input.txt | head -n 10000
0/3/3032.jpg
0/3/3034.jpg
  • -F$'\t' sets the field separator to TAB
  • $2 = /helicopter/ matches only on field 2
  • print $3 prints field 3 on matches

If you want case-insensitive full-word matches, try this:

awk -F$'\t' 'tolower($2) ~ /\<helicopter\>/ { print $3}' input.txt | head -n 10000

The \< and \> (word boundary markers) probably only work on gawk - which is standard if you're on Linux. And note also that the comparison operator has changed from = to ~.

  • A-W-E-S-O-M-E-!-!-! – Atcold Oct 26 '15 at 3:12
  • Next thing in my todo list: read man awk. – Atcold Oct 26 '15 at 3:12
  • the awk man page is a reference only, won't teach you how to use it. search for awk on this site instead. lots of good awk examples and explanations here. – cas Oct 26 '15 at 3:13
  • With grep I specified w, i.e. match full word (not only parts). Moreover, I would also like it to be case-insensitive. Could you add this two things? – Atcold Oct 26 '15 at 3:14
  • btw, not at all awesome...this is about as absurdly trivial an example of awk as you can get. – cas Oct 26 '15 at 3:15
2

You might want to also try locally setting LC_ALL=C in your environment.

If LC_ALL is a UTF8 locale, then that can imply that grep needs to do UTF8 decoding on the input stream before doing matching, and that can significantly slow down grep

  • This is more a comment that an answer, is it? – Atcold Oct 29 '15 at 5:08
  • @Atcold OP had a performance problem with grep. This is one way to reduce performance problems with grep. – Kent Fredric Oct 29 '15 at 5:12
  • It's not a solution to every problem, but answers typically aren't, additionally, if some other person has related issues to yours, this answer might be all they needed. – Kent Fredric Oct 29 '15 at 5:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.