3

I have a set of data in a file.

psf7433-nlhrms
unit7433-nobody
unit7333-opera
bpx7333-operations
app7333-osm
unit7330-partners
psf7331-pdesmond
unit7333-projm
mnp7330-redirect
unit7333-retailbanking
cpq7333-rkarmer
unit6333-sales
ring7323-support
unit7133-telco
post7323-uadb
sun7335-ukhrms
burp7133-wfnmreply

How to ignore the starting alphabetic characters in each line and the characters after the numeric and get the count of the unique numbers. (or) How to retrieve only the numeric value in each line and get their unique count.

Considering we manage to extract only the numeric values, we will get this.

7433
7433
7333
7333
7333
7330
7331
7333
7330
7333
7333
6333
7323
7133
7323
7335
7133

Now, I want the unique count of the retrieved numeric values. So ignoring the repetitions, I should get the following final output.

8

I am unable to do this either by using awk or sed or even simple grep | cut

I do not want the list of extracted values, I want only the final count as the answer.

Help me!

1
  • 1
    I only get 8 unique values.
    – muru
    Dec 9, 2014 at 15:10

5 Answers 5

6

With grep, filter out just the numbers:

grep -Eo '[0-9]+-' file | sort -u | wc -l
  • [0-9] Matches any character between 0 and 9 (any digit).
  • + in extended regular expressions stands for at least one character (that's why the -E option is used with grep). So [0-9]+- matches one or more digits, followed by -.
  • -o only prints the part that matched your pattern, so given input abcd23-gf56, grep will only print 23-.
  • sort -u sorts and filters unique entries (due to -u), and wc -l counts the number of lines in input (hence, the number of unique entries).
19
  • i'm new to shell scripting so can you explain what this script does in short. thanks a ton!
    – Karthik
    Dec 9, 2014 at 15:15
  • as per my requirement, the script should ignore any numeric value after the '-' character in the input (if any). For eg: if the input is sun7335-ukhrms8768, it should not read 8768 as it falls after the '-' character.
    – Karthik
    Dec 9, 2014 at 15:23
  • @Karthik Does the - appear only once per line? And does it always appear after the first number?
    – muru
    Dec 9, 2014 at 15:24
  • yes only once .
    – Karthik
    Dec 9, 2014 at 15:25
  • 1
    @mikeserv presence of multiple numbers might be a problem. See the second comment on this answer.
    – muru
    Dec 10, 2014 at 0:34
1

You can use:

tr -dc '\-0-9\n' | sort -u -t- -nk1,1 | grep -c .

...which is, admittedly, more than a little inspired by muru's answer here. Differently, though, I use grep to count the lines rather than wc in case there are blank lines in input. His answer doesn't have a blank line problem as grep -o will only print lines with their match (as grep -c only counts them here), but tr does print blank lines because the newline is one of the few characters it does not delete. This means any number of blank lines in input would skew wc's results by one.

So tr here is more efficient probably than is grep -o but likely wc beats grep in the counting department. I like it this way I think for portability reasons, and also because I usually try to prune data with the most efficient filter first, and to use the less efficient ones later in the chain.

This lets sort pick the bits per line that it will consider in its -unique sort based on its -numeric sort -key which it splits on its -t-ab delimiter. tr -deletes the -complement of any numeric, dash, or newline byte in its input. That way - so long as there are no -dashes occurring before the numeric strings you wish to compare, then the only thing remaining to any line is:

#nothing at all

...or...

[numbers]

...or...

[numbers]-[more numbers]more-dashes-...

So when the output is piped to sort we instruct it only to compare numeric strings occurring before a dash if any. In that way -dashes or not - the only numbers which matter are the ones you want to count.

So then we grep -count lines containing at least a single . character. The following command prints 8:

tr -dc '\-0-9\n' <<\IN | sort -u -t- -nk1,1 | grep -c .
psf7433-nlhrms
unit7433-nobody
unit7333-opera
bpx7333-operations
app7333-osm
unit7330-partners
psf7331-pdesmond
unit7333-pro-09-0jm
mnp7330-redir09o-0ect
unit7333-retailbanking
cpq7333-rkarmer
unit6333-sales
ring7323-support


unit7133-telco
post7323-uadb
sun7335-ukhrms
burp7133-wfnmreply
IN
1

Use extended grep to and look for four digits, telling grep to only list the matches (as opposed to the whole line, which is the default):

grep -Eo '[0-9]+' <filename>

Sort this list of numbers and only output unique ones:

sort -u

Count the number of lines:

wc -l

Put it all together:

$ grep -Eo '[0-9]+' filename | sort -u | wc -l
8
3
  • I provided only 4 digit numerics in the sample data. My original data contains entries with 2,3 digit numeric values too. The only condition is it should not count any numeric values after the '-' character in the entry.
    – Karthik
    Dec 10, 2014 at 11:04
  • @Karthik - I've changed the grep search pattern to look for at least one digit. Dec 10, 2014 at 11:08
  • My input has a few entries which contains a numeric character after the '-' character. eg: unit7330-partners1. In such case, it should ignore any digits after the '-' character in the entry.
    – Karthik
    Dec 10, 2014 at 11:15
0

With , without any shell pipes (quicker) :

$ perl -lne '/\d+-/ and $h{$&}++;END{print scalar keys %h}' file
7
  • Hi, I just modified the question with a new set of input. Will the code be applicable for the new input too.
    – Karthik
    Dec 9, 2014 at 15:08
  • POST edited accordingly Dec 9, 2014 at 15:09
  • gives me the answer 0
    – Karthik
    Dec 9, 2014 at 15:16
  • You have to modify file by your own file Karthik Dec 9, 2014 at 15:17
  • 1
    New requirement satisfied chief Dec 9, 2014 at 15:24
0

An awk solution

awk -F'-' '{sub(/[^[:digit:]]+/, "", $1); a[$1]} END{for (k in a) ++i; print i}' file
8
1
  • thanks but will this script also read any numeric (just in case) which occurs after the '-' character.
    – Karthik
    Dec 10, 2014 at 6:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.