1

I am new in unix, I have a question regarding data subsetting, I would appreciate anyone's help. I have 23G input file containing millions of rows, but I want to keep only those rows where the first and 4th column are identical (scaffold names). here is the first few rows of my data set:

tscaffold94_798049_802097   999 NA tscaffold94_798049_802097   999 NA 1
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1029 NA 1
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1044 NA -0.0463767871013283
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1045 NA -0.939576278422824
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1130 NA -0.0831304705346077
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1180 NA -0.931681175211672
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1187 NA -0.940010336852543
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1202 NA 1
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1224 NA 1
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1269 NA 1
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1313 NA -0.201478578143067
tscaffold94_798049_802097   999 NA tscaffold94_798049_802097  1384 NA 1
tscaffold94_798049_802097   999 NA tscaffold94_878564_884314  3259 NA -0.595441932439136
tscaffold94_798049_802097   999 NA tscaffold94_878564_884314  3304 NA 0.745699172241005
tscaffold94_798049_802097   999 NA tscaffold94_878564_884314  3319 NA -0.570318634275133
tscaffold94_798049_802097   999 NA tscaffold94_878564_884314  3588 NA -0.60363963711489
1

awk is your friend in this case; the columns become variables in the awk script, and so it's easy to check for equivalence (or not), and perform an action like print (the current row is implied)

awk '{if($1 == $4) print}' < input

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.