4

I am watching log traffic and see regular abuse of the various image search engines, Bing particularly.

Example URL:

http://www.bing.com/images/search?q=dagger+genesis+solo&view=detailv2&&&id=C65E811DFE01FB11258D2EB4F516F3DD8F09049C&selectedIndex=4&ccid=ffC0NVO8&simid=608046582336849763&thid=JN.XnLfF7qiZGwjJzTCR6f7ZQ&mode=overlay

I want to pull out the search itself, the "dagger genesis solo".

I can

grep -o '=*' 

but that leaves the = sign and everything that follows the search.

I want to grab everything between "search?q=" and the first "&" the end of the last word preceded by a "+".

I could get there in a very long and convoluted way using awk or cut to get strip as much of my string away as possible, and then use specified field separators to put each word in its own column and then print only those columns. But even that method wouldn't be consistent as the searches can be any length and include just about any character.

I am thinking there is a much easier way though. Ideas?

My final goal is to strip the search queries and collate them into unique entries.

3

You can do

sed 's/^.*search?q=\([^&]*\)&.*/\1/' file

What this does is does a non greedy match between the search?q= and the &

Which outputs

dagger+genesis+solo

If you want to replace the + signs with spaces,

sed 's/^.*search?q=\([^&]*\)&.*/\1/;s/+/ /g' file

Which outputs

dagger genesis solo
  • The second option works perfectly as a one stop option instead of me needing to take care of the +'s as a post-action. Could you kindly explain how the brackets inside the parenthesis works with sed? – user112802 Jun 5 '15 at 3:05
  • 1
    The [^&] means match a single character that is not the ampersand. The way to match a character is [], and the ^ is the negation operator. The * means match any number of occurrences of the previous character. Also the second option is just the first option combined with an additional command to substitute the + signs, namely s/+/ /g – bkmoney Jun 5 '15 at 3:08
4

If your system's grep supports PCRE-mode, you could use lookarounds (zero length assertions) to select characters between search?q= and &

grep -Po '(?<=search\?q=).+?(?=&)'

Using the non-greedy modifier ? in between makes the match stop at the first &.

  • That works as I originally wanted, thank you. – user112802 Jun 5 '15 at 3:06
2

With sed:

sed 's/\([^=&]*.\)\{2\}&.*/\1/' <<""
http://www.bing.com/images/search?q=dagger+genesis+solo&view=detailv2&&&id=C65E811DFE01FB11258D2EB4F516F3DD8F09049C&selectedIndex=4&ccid=ffC0NVO8&simid=

When an occurrence count is specified for a back-referenced match, sed should reference only the specified match. So in the above example the reference returns only

dagger+genesis+solo
1

Using grep look-behind and look-ahead assertions:

grep -oP "\=\K.*(?=\&view)"

Here,

\K          ==>  zero-width look-behind assertion
(?=\&view)  ==>  zero-width look-ahead assertion

Hence, only the part in-between \= and &view i.e. .* is printed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.