0

Input file

1 2 3 1 4 5 6 1 1 2 34 5 6 2

I want output like this

1 2 3 4 5 6 34 

( all the duplicate number should print only once )

closed as unclear what you're asking by Kusalananda, Anthony Geoghegan, Jesse_b, slm Jul 13 '18 at 14:42

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    Can the output be sorted, or does it have to keep the same order? – Alexander Jul 12 '18 at 6:24
  • 1
    Can you please clarify your Q? It's unclear whether sorting the results is OK or not. You have both types of A'ers ATM. – slm Jul 12 '18 at 14:12
  • 2
    Does your input always consist of a single line? – NickD Jul 12 '18 at 19:25
0

With awk (to remove duplicates from whole file):

awk '{ SEP=""; i=0;  
       while (++i<=NF) { printf (!unq[$i]++? SEP $i:""); SEP=" "};
       print "";
}' infile

or remove duplicates from each line separately:

awk '{ SEP=""; i=0; delete unq;
       while (++i<=NF) { printf (!unq[$i]++? SEP $i:""); SEP=" "};
       print "";
}' infile

Loop over numbers and if it was not previously seen, add it to the array called unq and printf it else print nothing.

future reading:

3

Perl to the rescue!

perl -lane 'print join " ", grep ! $seen{$_}++, @F' < file
  • -l removes newlines from input and adds them to output
  • -n processes the input line by line
  • -a splits each input on whitespace into the @F array

The @F array is iterated over by grep, keeping only the elements that are seen for the first time (i.e. their value in the %seen hash are zero, as a negation of zero is true). join just glues them back together.

  • ... or perl -MList::MoreUtils=uniq -alne 'print join " ", uniq @F' – steeldriver Jul 12 '18 at 11:45
  • 1
    @steeldriver: List::Util now provides uniq, too. – choroba Jul 12 '18 at 11:59
  • Ah thanks - I thought it did but I was testing on an old box with 5.22 – steeldriver Jul 12 '18 at 12:28
2

NOTE: This solution keeps the results in their original order!


Easy way to do this using awk and tr. If your content is in a file, FILE:

$ tr ' ' '\n' < FILE | awk '!x[$0]++' | tr '\n' ' '

Example

$ echo "1 2 3 1 4 5 6 1 1 2 34 5 6 2" | \
    tr ' ' '\n' | awk '!x[$0]++' | tr '\n' ' '
1 2 3 4 5 6 34

How it works

  • tr ' ' '\n' - switches spaces to newlines
  • awk '!x[$0]++' - prints characters that do not repeat
  • tr '\n' ' ' - switches newlines back to spaces
  • This assumes that the input is always a single line, right? That's what the OP described in the question, but it is not clear to me that that was the intent. – NickD Jul 12 '18 at 19:27
  • @NickD - This can deal with the data both ways. I agree though, that the OP needs to clarify several questions that many of us have either assumed or didn't realize to ask. – slm Jul 12 '18 at 19:49
  • Don't need tr just awk -vRS='[ \n]' -vORS=' ' '!x[$0]++' and optionally add ;END{printf "\n"} if you want a well-formed line out. – dave_thompson_085 Jul 13 '18 at 8:30
0

You can use tr, awk or sed to get all numbers in separate line, then use sort and uniq to remove duplicate entries

Example with tr:

$ tr " " "\n"  < <INPUT-FILE> | sort -u | tr "\n" " "
  • The redir was intentional 8-) – slm Jul 12 '18 at 6:38
0

You can do this with sort. Sort only operates on rows so you need to replace the spaces in your file with new lines.

Example if file test.txt contains:

 1 2 3 1 4 5 6 1 1 2 34 5 6 2

The following code will return what you want

>sed "s/ /\n/g" test.txt | sort -un
1
2
3
4
5
6
34

Note that while sort -u will return "unique" (non-duplicated values), the are ordered alphabetically unless you sort numerically with sort -n. Sort is necessary here as uniq only removes consecutive duplicates so you would need to pipe sort -n | uniq and sort anyway.

0

A simple Unix style way of accomplishing this is to use the basic core utilities and pipe the contents of test.txt to the appropriate program rather than use a full-blown scripting language, like so:

$ cat test.txt | tr ' ' '\n' | sort -u | tr '\n' ' '
  • Nice. Same concept with different utilities (requires bash): (set -f; printf "%s\n" $(<file) | sort -un | paste -sd " ") – glenn jackman Jul 12 '18 at 16:47
  • As of late, since I've been experimenting with various flavors of BSD, I try to avoid using bashisms unless necessary. After all, why add an additional dependency if it's not necessary. (See Also: Shellshock) – jaywilliams Jul 13 '18 at 14:23
0

In bash : Preserves the reading order. Reads several lines in the file. Returns a single line.

 unset d
 while read r; do
  [ "$d" ] || d=${r/ */}
  for s in $r; do
    for s2 in $d; do
      if [ "$s" = $s2 ]; then
        s=
        break
      fi
    done
    d="$d $s"
  done
done < file
echo ${d}
unset d r s s2

A script is better

echo '
#!/bin/bash

if [ $# = 0 ]; then
  echo "${0##*/} <filepath>" >&2
  exit
elif [ ! -f $1 ]; then
  echo " Error : $1 is not a file." >&2
  exit 1
fi

while read r; do
  [ "$d" ] || d=${r/ */}
  for s in $r; do
    for s2 in $d; do
      if [ "$s" = $s2 ]; then
        s=
        break
      fi
    done
    d="$d $s"
  done
done < $1
echo ${d}
' > ~/uniq-word

For use it :

bash ~/uniq-word <filepath>

Not the answer you're looking for? Browse other questions tagged or ask your own question.