2

update Sample data file is below:

empid;1001
empname;ABC
salary;3000
dept;ABC
age;24
dept;112
JOD;20170101
empid;A2001
salary;5000
dept;XYZ
age;27
JOD;20170303
age;92
empid;1002
empname;MAN
salary;11000
dept;SCI
age;30
dept;Geology
JOD;20180607
empid;1005
empname;NAME
salary;10200d
dept;XYZ
JOD;20161212

I need to search for all attributes and copy the first occurrence of each to a nother file. output should look like :

empid;1001
empname;ABC
salary;3000
dept;ABC
age;24
JOD;20170101
empid;2001
salary;5000
dept;XYZ
age;27
JOD;20170303
empid;1002
empname;MAN
salary;11000
dept;SCI
age;30
JOD;20180607
empid;1005
empname;NAME
salary;10200
dept;XYZ
JOD;20161212

2nd occurrence of dept should not be considered if in each set of values

empid,empname,salary,dept,age,JOD.

CURRENLTY i am using below code:

awk -v FS=';' OFS=';'{ 
if ($1 == "empid" || $1 == "empname" || $1 == "salary" || $1 == "dept" || $1 == "age" || $1 == "JOD" ) print $0 }' FILE_NAME > NEW_FILE_NAME.

but its taking second occurrence of dept also. Please guide me through it.

4
  • 1
    It looks as if the example data is basically four concatenated records, is that correct? Each record starts with the empid attribute, and ends with the JOD attribute? – Kusalananda May 8 '20 at 9:46
  • Hi @kusalananda, this is column based data, that means attributes are coming in 1st field and 2nd fields are having respective values.. they are not conctenated. May be its was a issue while question was posted. – Naresh May 8 '20 at 11:25
  • Sure, I can see there are two columns. But the data seems to be made up of four sets of rows, each set of rows constituting the entry for an employee (a "record" for that person). – Kusalananda May 8 '20 at 11:38
  • yes that's correct @Kusalananda, i Thought you all are seeing the sample data in a text /line/row format. But they are actually 6 set of values as i have mentioned: EMPID EMPNAME SALARY DEPT AGE JOD. there can be a possibility that data for some attribute is missing from the file. – Naresh May 8 '20 at 11:41
4

Assuming that Kusalananda is right, and that each employee record starts with the line empid, the following awk command should work:

awk -F';' '$1=="empid" {delete a} !a[$1]++' input.txt > output.txt

This uses an array variable a to track which of the attribute names was already encountered, and print the current line only if that was not yet the case. The array is reset every time the empid attribute is encountered.

For a more in-depth explanation:

  • $1=="empid" {delete a} will delete the array a every time a new record starts
  • !a[$1]++ makes use of the awk shorthand notation that a 1 outside of a conditional rule means "print the line", whereas a 0 would mean "don't print".
  • The a[$1]++ will increase an "occurence counter" for every value of the attribute name, which here is taken as "array index".
  • The evaluation !a[$1]++ will first check if the current value of the array entry is zero (i.e. the attribute was not yet encountered), perform the print action if true (thanks to the negation operator), and increase the counter afterwards (this works the same as prefix/postfix increment in C-style programming languages). Thus, if the attribute was not yet encountered, it is printed, but later occurences are ignored.

Note that while the delete a statement conforms to a syntax that was accepted to POSIX standard in 2012, and the above works on GNU awk, mawk and nawk, Stéphane Chazelas pointed out that for those implementations that don't support this syntax, the

delete a

should be replaced by

split("",a)
11
  • 1
    Clever and shorter than what I was working on. Setting OFS is not needed though, so -F ';' would be enough. – Kusalananda May 8 '20 at 9:57
  • To increase portability, you can replace delete a with split("", a). – Stéphane Chazelas May 8 '20 at 11:24
  • thanks you @AdminBee, but the data is in Column format, means 1st column is having attnames and 2nd field/columns is having there respective values. it may have been shown in a different way while question was uploaded. Please look at the question once again . – Naresh May 8 '20 at 11:32
  • @Naresh I am usure if I correctly understand your comment.How does the output of my suggested awk command differ from the desired output as stated in your question? I tested it with your sample input copy-and-pasted into a file, and the output looked just like you stated under "output should look like". – AdminBee May 8 '20 at 11:34
  • 1
    @Naresh ok, this is of course a game changer. In order to keep this question thread as useful and understandable as possible for a wide range of people who may be facing a similar problem, I would ask you to undo your last edit and then either consider this question solved and ask a new question with the "additional" constraint, or edit your current question by adding an "Addendum" or "P.S." (or however you like to call it) which then describes the new challenge separately. In the meantime: are the "surplus" empid lines guaranteed to directly follow the first one of every record? – AdminBee May 8 '20 at 13:38
1

This is the same basic idea as AdminBee's solution, but slightly less elegant (I store all values in memory for no good reason), albeit slightly shorter:

gawk -F';' '$1=="empid"{i=$2} ++a[i][$1]==1' file

We set i to the employee ID if the first field is empid. Then, we take advantage of a nice little trick in awk: when an expression evaluates to true, awk will print the line. So, a[i][$1] is an element of a two dimensional array, whose first key is the current empid (stored as i) and whose second key is the 1st field of the current line (a[i][$1]). Since ++ adds one to this, the expression ++a[i][$1]==1 will be true only for the first time each field is seen for a particular empid. Since we only print if it is true, the command will print the first occurrence for each id.

Note that this requires GNU awk.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.