1

I have 31 files in one directory. File format is as below :

TimeStamop      user-id
1471417712595   1936f35c-XXXX-XXXX-XXXX-49898523b8ea
1471417717238   e3153e8d-XXXX-XXXX-XXXX-7710089c77bb
1471417718730   4e470320-XXXX-XXXX-XXXX-29bc448f09a1
1471417719211   e8b6a6c0-XXXX-XXXX-XXXX-dd68d836fb31
1471417723775   cbc009c1-XXXX-XXXX-XXXX-88cda2b1b812
1471417723775   cbc009c1-XXXX-XXXX-XXXX-88cda2b1b812
1471417725790   c332eba7-XXXX-XXXX-XXXX-509c42a621d1
1471417725522   4712acf1-XXXX-XXXX-XXXX-11a24bcbe7c8
1471417601920   502bc0dd-XXXX-XXXX-XXXX-eff38b468461
1471417604615   8c6b2ef9-XXXX-XXXX-XXXX-9efb1223e383

I just want timestamp information from it and want to write all the timestamps in one file using linux shell script. Total file size (31 files size) is of 4.5 GB. I want efficient method for doing this task. I have done this in python. but want it in shell script.

2
  • What format of timestamp is that ? It doesn't appear to be the usual 'seconds since 1970'. Converting it using the normal perl approach returns a year far, far into the future. $ perl -e 'print scalar localtime(1471417717238),"\n"' Mon May 15 15:40:38 48597. gnu date also far into future $ date -d @1471417712595 Mon May 15 14:23:15 UTC 48597
    – steve
    Oct 23, 2016 at 21:03
  • 1
    this is timestamp in milliseconds. @steve Oct 23, 2016 at 21:23

1 Answer 1

1

With awk, from the directory containing the files:

awk 'FNR>1 {print $1}' * >output.txt
  • FNR>1 matches all records with record number greater than 1 from each file i.e. this is to get rid of the header from each file

  • For those records, {print $1} prints the first field only which are the timestamps

3
  • 1
    awk '$0^=1' * >output.txt also works.
    – steve
    Oct 23, 2016 at 21:24
  • is there any way to unique all the timestamps and sort it ? Oct 23, 2016 at 21:34
  • 1
    @KrunalParmar yes, do: awk 'FNR>1 {print $1}' * | sort -u
    – heemayl
    Oct 23, 2016 at 21:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .