I have a file that contains records with an identifier. For each identifier, there may be multiple records. I want to search all records with the same identifier and look for a particular pattern (Y at specific position) in those records; if the pattern exists, I want to update the first record for that indicator with that pattern. How can I best accomplish that with scripting? (Unix or windows). The file is already sorted by identifier.
Here is a sample of what I want to accomplish:


if, one of the records for identifier1 has a 'Y' at position 14, then write that 'Y' to the first occurrence of the identifier1 record, i.e.,


I am not sure what tool (awk, grep, sed) would be best suited for this? Any idea how to tackle it?

  • Are all the records the same length? Are there records other than identifier1?
    – Scottie H
    Oct 24 '19 at 16:28
  • all records are the same length. There will be many different identifiers in the file and for each identifier, there may be multiple records.
    – user378840
    Oct 24 '19 at 16:33
  • Is each identifier 11 characters long? Or how do we know where an identifier ends?
    – Freddy
    Oct 24 '19 at 17:10

Using awk and reading the input file twice to process it.

This is under the assumption that your identifier is 11 characters long and that you're looking for lines with Y as the 15th character (as in your example). If the identifier is not always 11 characters long, then the first line of the script needs to be modified.

First pass: Save the first record for each identifier in an array and modify this array element if a record with Y is found.

Second pass: Replace the line of the first record for each identifier with the saved and possibly modified array value and print the line.

awk '{
  ident=substr($0,0,11)  # get identifier
  if (NR==FNR){          # first pass
    if (!(ident in a)){  # if identifier is not present in array
      a[ident]=$0        # save current line in array
    if (substr($0,15,1) == "Y"){  # if `Y` is found in current line
                                  # replace character with `Y` in array value
  else {               # second pass
    if (ident in a){   # if identifier is present in array
      $0=a[ident]      # replace current line
      delete a[ident]  # delete array element
    print              # print current line
}' file file

NOT elegant. For informational purposes only.

First, identify all of the identifier1 records
command1: grep '^identifier1' my_file_name
This will print only the lines with your chosen tag. Later, you can look for records with a different identifier.

Once you have these records, you can search just column 14, with the cut command:
command2: _command1_ | cut -c14
For clarity, I'm using shorthand notation on the commands. Hope they makes sense to you.
command2 uses the cut command to only print the 14th character (-c for Character).

You can then look for 'Y' in this output.
command3: _command2_ | grep -q Y The -q says "Quiet" -- don't output anything.
rc=$? Note: Some say you don't need rc, just look at $?. Your call. At this point, if $rc (or $?) is 0, grep found at least one upper case Y. Otherwise, it will be 1 (other numbers are possible, indicating an error).

Let's code that this way:
if [ $rc -ne 0 ] then echo "No Ys found" exit fi
Note: if you place ANY commands between the grep -q Y and the if, you MUST use the rc=$? Otherwise $? is fine.

To change ALL of the indicators for N to Y, you would run this sed command:

sed -i "/^identifier1/ s/\(………….\).\(.*\)/\1Y\2/" my_file_name

The -i says In-place editing of the file.
the /^identifier1/ says only match lines starting with indetifier1
the s means search and replace. The SearchFor is after the / The Replace With is after the next slash.
SearchFor: You want the 14th character. There is an elegant way to do this in sed. I'll leave that for other folks to explain. The simple answer is:
-dot- means any character. So 13 dots means match the first 13 characters. Because we want to keep these in-tact, we'll store them in a bucket. The is it ( and ) -- escaped parenthesis. So:
/\( 13 dots \) . \( dot * \) say grab the first 13 characters of the line and store them in Bucket #1. the dot means match 1 character. the dot star says "Match 0 or more characters" -- i.e. everything else -- and store it in Bucket #2.

Now for the replace:
/\1Y\2/ says Replace the line with the contents of Bucket #1, followed by Y and then Bucket #2.

There is a way to clean this up and make it more efficient. I don't a Linux box in front of me and can't remember off the top of my head how to do it.
When some other FAB folks post it up, you ought to be able to understand it now.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.