0

I have a huge file of XML data containing the lines like

<fonts> some of the data </fonts>
<fonts> some of the data </fonts>
<fonts> some of
 the data </fonts>
<fonts> some of the data </fonts>
<fonts> some of the data </fonts>

In between I am getting the newline character...

$>uname -a
SunOS ******* 5.11 SunOS_Development
  • 3
    Use an xml parser? – Thor Nov 6 at 10:55
1

Given an XML file file.xml:

<?xml version="1.0"?>
<root>
  <fonts> some of the data </fonts>
  <fonts> some of the data </fonts>
  <fonts> some of
the data </fonts>
  <fonts> some of the data </fonts>
  <fonts> some of the data </fonts>
</root>

You may apply the normalize-spaces XPath function to all fonts nodes using XMLStarlet:

$ xmlstarlet ed -u '//fonts' -x 'normalize-space()' file.xml
<?xml version="1.0"?>
<root>
  <fonts>some of the data</fonts>
  <fonts>some of the data</fonts>
  <fonts>some of the data</fonts>
  <fonts>some of the data</fonts>
  <fonts>some of the data</fonts>
</root>

This removes flanking whitespace and replaces all runs other types of blanks with single spaces in all fonts nodes.

Would you want to only remove newlines from the fonts nodes' data:

$ xmlstarlet ed -u '//fonts' -x 'translate(., "'$'\n''", "")' file.xml
<?xml version="1.0"?>
<root>
  <fonts> some of the data </fonts>
  <fonts> some of the data </fonts>
  <fonts> some of the data </fonts>
  <fonts> some of the data </fonts>
  <fonts> some of the data </fonts>
</root>

This relies on your shell expanding $'\n' to a literal newline character.

  • xmlstarlet -ksh: xmlstarlet: not found [No such file or directory] – user380754 Nov 6 at 13:10
  • also i should not get the <?xml version="1.0"?> <root> these tags in the files – user380754 Nov 6 at 13:10
  • @user380754 You would have to install xmlstarlet separately (I don't know how to do that on your SunOS system). I added dummy tags to the example data as the XML that you showed was not a complete document. The tags that I added are not important for anything other than to make the XML document well formed. – Kusalananda Nov 6 at 14:11
0

The standard method would be the tr command. XML can have (and usually does) extremely long lines, because whitespace is not really part of the XML specification. Most Unix text commands are line-based, and very long lines are likely to break things like sed or awk (on Linux they tend just to be a huge performance hit). tr does not have to care about lines at all, so it can handle this data well.

tr -d '[\r\n]' <inFile > outFile

However, I am concerned about your third "fonts" line, which you show as split. Taking out that newline will run the texts together without any whitespace. I know that Excel .xlsx files use newlines (and carriage returns) where there is multi-line text in a cell.

Why do you need to remove the whitespace? It should not cause any problems. In fact, I normally run XML through "xmllint --format", because that makes it viewable in editors. It reads back in Excel just fine: I have pretty-printed XML, imported it to Excel, written it back out as long lines, and done a cksum, and it was identical.

You can avoid the long-line problem in awk, by defining RS = ">"; There are always enough > in XML to avoid long lines. awk then sees each XML entity on a separate line, with at most one text item before it. Where there are genuine newlines, they will stay in the text, so you would see your third input as

<fonts>
some of\nthe data</fonts>

You can then gsub each NL or CR/LF to a SPACE and preserve the wording.

If you need that code in more detail, ask.

In case you wonder what Excel has to do with XML: an Excel file (.xlsx, .xlsm etc, not not .xls) is just a zip of a bunch of 20 or more XML files.

  • if we execute tr -d '[\r\n]' <inFile > outFile then the output will be continious <fonts>some of the data</fonts><fonts>some of the data</fonts> i want the output to be like <fonts>some of the data</fonts> <fonts>some of the data</fonts> – user380754 Nov 6 at 13:13
  • That's why you need something complex enough to recognise XML syntax, at least to the extent of <fonts> being an opener and </fonts> being the closer, and in fact <fonts myAttr /> being inline. xmllint --format will do that, but your input is not valid XML because you don't like <xml? identity tag. I offered to post an awk one-off but you don't ask for it, or tell me what OS and awk version you have. – Paul_Pedant Nov 9 at 11:19
0

We can use sed 's/\n/ /g' notice the blank space after the second forward slash, the above expression substitutes all newline characters by a whitespace

0

We cannot use sed s'/\n/ /g'

sed works on new-line separated text. When each item of input gets run through the expressions, it has already had the newline removed, so the expression will never match. By definition, you can never get a newline in the middle of a line. (Tested.)

In awk, you can switch the RS 'end-of-line' character so some other character is used, and then newlines will be left alone. Or (in gawk) you can make RS empty, and then the whole file will come in as one long line. I can't think of any other standard commands that let you do this.

To handle newlines in sed, you need to use the holding space, and figure how to stitch that onto the next line according to how the end/start conditions match up (like, both text, tag follows text, etc).

Also, the OP is on SunOS 5.11, and I would bet dollars to doughnuts that it has a line length restiction of something like 2048 chars. So generic XML will break it. Solaris awk breaks on 6144 (6K) chars.

I am not even sure this file will be proper XML. The OP says no to the required (missing, presumed dead) tag, so there may be other home-brew issues lurking too.

Not the answer you're looking for? Browse other questions tagged or ask your own question.