16

I am running the following 2 sed commands. The first one adds newline characters where I want them, the second also adds newline characters where I want them, BUT also adds an extra one at the end of the file where there wasn't one before.

sed -e 's|\<LIST_G_STATEMENT>|&\
|g' ${XMLDIR}/statement_tmp_1.xml > ${XMLDIR}/statement_tmp_2.xml

sed -e 's|\</LIST_G_STATEMENT>|&\
|g' ${XMLDIR}/statement_tmp_2.xml > ${XMLDIR}/statement_tmp_3.xml

Using od -c on all 3 of the files gives the following output.

statement_tmp_1.xml (no \n at end of file)

1314700    T   A   T   E   M   E   N   T   >   <   /   L   I   S   T   _
1314720    G   _   S   T   A   T   E   M   E   N   T   >   <   /   G   _
1314740    S   E   T   U   P   >   <   /   L   I   S   T   _   G   _   S
1314760    E   T   U   P   >   <   /   A   R   X   S   G   P   O   >
1314777

statement_tmp_2.xml (no \n at end of file)

1314700    S   T   A   T   E   M   E   N   T   >   <   /   L   I   S   T
1314720    _   G   _   S   T   A   T   E   M   E   N   T   >   <   /   G
1314740    _   S   E   T   U   P   >   <   /   L   I   S   T   _   G   _
1314760    S   E   T   U   P   >   <   /   A   R   X   S   G   P   O   >
1315000

statement_tmp_3.xml (\n at end of file - where did it come from?)

1314700    S   T   A   T   E   M   E   N   T   >   <   /   L   I   S   T
1314720    _   G   _   S   T   A   T   E   M   E   N   T   >  \n   <   /
1314740    G   _   S   E   T   U   P   >   <   /   L   I   S   T   _   G
1314760    _   S   E   T   U   P   >   <   /   A   R   X   S   G   P   O
1315000    >  \n
1315002

I am running AIX 5.3

Basically, I either want it to stop adding the extra \n, or find a way of removing it.

  • Just a question: why are you using a literal newline in your substitution pattern when you could have used s|...|&\n| just as well? – Joseph R. Nov 4 '13 at 10:59
  • @JosephR. \n in the right hand side is not portable. – Stéphane Chazelas Nov 4 '13 at 11:03
  • @StephaneChazelas That's weird. Is it a CR vs CRLF thing? – Joseph R. Nov 4 '13 at 11:05
  • 2
    A file which doesn't end in a newline character is not a text file, so the behaviour with text utilities on them is unspecified. Use perl or other tool that can deal with binary data. – Stéphane Chazelas Nov 4 '13 at 11:06
  • 4
    @JosephR. No, \<LF> is the traditional and POSIX way to add a LF character. \n would typically substitute a n character in anything but GNU sed. – Stéphane Chazelas Nov 4 '13 at 11:07
9

You should consider yourself lucky that AIX sed added that missing newline characters.

A non-empty file that doesn't end in a newline character is not a text file (at least as per the POSIX definition of a text file) as a text file is meant to contain lines and lines are a (not-too-long) sequence of characters terminated by a newline character, so the behaviour of text utilities like sed on it is unspecified and in practice varies from implementation to implementation.

Some sed implementation would have dismissed those spurious character after the last line.

AFAIK, xml files are meant to be text files, so that means sed just fixed it for you.

If you do need that file not to end in a newline character, then you could use perl or other tools that can cope with non-text data.

perl -pe 's|<LIST_G_STATEMENT>|$&\n|g'
0

Here's a way to remove the final newline from a file using dd:

printf "" | dd  of='/path/to/file' seek=<filesize_in_bytes - 1> bs=1 count=1

To test whether a file ends with a newline you could use:

tail -c 1 /path/to/file | tr -dc '\n' | wc -c

And to get the file size in bytes use:

wc -c < /path/to/file
0

According to this AIX manual IBM's tail does -reverse - which looks pretty cool. So long as your file is under 20KB the following should work:

tail -r <file | dd bs=1 skip=1 | tail -r >file.new

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.