5

I have several hundred .xhtml files in a sub-directory(*) and I want to delete all DIVs with a specific class (and the entire contents of those DIVs - including other divs, spans, image and paragraph elements) from them. The DIV may appear zero, one, or more times at any arbitrary depth within each .xhtml file.

The specific DIVs I want to delete are:

<div class="portlet solid author-note-portlet">.....</div>

Using the xml_grep utility from the perl XML::Twig module, I can run xml_grep -v 'div[@class="portlet solid author-note-portlet"]' file*.xhtml and it will remove all instances of that div from the .xhtml files and display the result on stdout. Exactly what I want, except for "display on stdout".

If xml_grep had some kind of in-place edit option, that would be fine, I'd just use that....but it doesn't, so I'd have to write a wrapper script that used a temporary file or sponge and run xml_grep against each .xhtml file individually, which would be slow and tedious. Or I could hack a copy of xml_grep so that it could edit its input file(s).

But I don't want to do either of these things, I want to use the existing tool which can already do this, I want to use xmlstarlet - it'll be faster, has in-place edit, and I won't have to run it once per filename.

The trouble is that no matter what I try (and I have tried dozens of variations), I cannot figure out the correct xpath specification to delete a div with this class. e.g. I have tried:

xmlstarlet ed -d "div[@class='portlet solid author-note-portlet']" file.xhtml

and (with different quoting)

xmlstarlet ed -d 'div[@class="portlet solid author-note-portlet"]' file.xhtml

and

xmlstarlet ed -d '//html/body/div/div/div[@class="portlet solid author-note-portlet"]'

and dozens of other variations. None of them have resulted in any change to the xhtml output. This is the point at which I usually give up on xmlstarlet and write a perl script, but this time I'm determined to do it with xmlstarlet.

So, what's the correct way to specify this div class for xmlstarlet?

BTW, for one example .xhtml file (with two instances of this div, which happen to be at the same depth...which is fairly typical but not universal), xmlstarlet el -v says:

$ xmlstarlet el -v OEBPS/file0007.xhtml | grep author-note-portlet
html/body/div/div[@class='portlet solid author-note-portlet']
html/body/div/div[@class='portlet solid author-note-portlet']

(*) Not that it matters, but these .xhtml files are inside a .epub file(**) generated by the FanFicFare plugin for Calibre - which downloads all chapters from books on various fiction web sites and turns them into an epub file (which is basically a zip archive containing XHTML and CSS files and maybe jpeg or gif files, along with a bunch of metadata files).

<div class="portlet solid author-note-portlet"> is used by one site (Royal Road) for authors to include a note with a chapter. Some authors use it sparingly, and insert short notes about either the chapter or the book or brief announcements about random stuff, with maybe a link to their patreon page...fine, no big deal.

Others use it to add a half page note with links to 10 of their other books at the start of each chapter and again to add three and half pages of links (with cover images) to those books at the end of each chapter. Which is kind of OK-ish if you're reading it in serial form chapter-by-chapter on the web site, but not if you're reading it as a book - ~4 pages of self-promotion for every 6-10 or so pages of story is excessive and distracting. And, BTW, that's 4 "pages" on my 10 inch android tablet - it's more than double that on my phone.

I can easily add display: none to the epub's style sheet for this class, but I want to actually delete the divs from the .xhtml files. They noticeably inflate the .epub file size.

(**) extracting the contents of the .epub with unzip and rebuilding it afterwards are way outside of the scope of this question, so please don't get distracted by irrelevant details. Already handled.


Sample .xhtml file, edited down to the bare minimum (and story/chapter/author name anonymised to protect the "guilty :-):

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Chapter Five - Chapter Name</title>
<link href="stylesheet.css" type="text/css" rel="stylesheet"/>
<meta name="chapterurl" content="https://www.royalroad.com/fiction/URL"/>
<meta name="chapterorigtitle" content="Chapter Five - Chapter Name"/>
<meta name="chaptertoctitle" content="Chapter Five - Chapter Name"/>
<meta name="chaptertitle" content="Chapter Five - Chapter Name"/>
</head>
<body class="fff_chapter">
<h3 class="fff_chapter_title">Chapter Five - Chapter Name</h3>
<div class="chapter-inner chapter-content"><div class="portlet solid author-note-portlet">
                    <div class="portlet-title">
                        <div class="caption">
                            <i class="fa fa-sticky-note"></i>
                            <span class="caption-subject bold uppercase">A note from Author Name</span>
                        </div>
                    </div>
                    <div class="portlet-body author-note"><p><span>About a dozen or so p, span, img, and br tags here</span></p>
</div>
                </div>
<p> story text here.  a few hundreds p, br, etc tags
</p>
            <div class="portlet solid author-note-portlet">
                    <div class="portlet-title">
                        <div class="caption">
                            <i class="fa fa-sticky-note"></i>
                            <span class="caption-subject bold uppercase">A note from Author Name</span>
                        </div>
                    </div>
                    <div class="portlet-body author-note"><p>several dozen more p, span, br, img, etc tags here</p>
</div>
                </div>
</div>
</body>
</html>

2 Answers 2

8

The correct way to do it with xmlstarlet is

xmlstarlet ed --inplace -N xmlns="http://www.w3.org/1999/xhtml" \
    --delete '//xmlns:div[@class="portlet solid author-note-portlet"]' file

or, using short options,

xmlstarlet ed -L -N xmlns="http://www.w3.org/1999/xhtml" \
    -d '//xmlns:div[@class="portlet solid author-note-portlet"]' file

Since the document uses a default namespace, we need to let xmlstarlet know that all nodes belong to this namespace and then also prefix the node's name with the namespace placeholder in the XPath expression.

According to the documentation, -N must be the last "global option", i.e. it must come after -L (another global option). The -d is the "delete operation" to xmlstarlet ed, so it's not one of the global options.

The XPath //xmlns:div will look recursively for a node called div in the xmlns namespace.

In the question, apart from not handling the namespace, you either under-specified or over-specified this. Using div, which is the same as /div, would be matching a root node, and //html/body/div/div/div would be matching an immediate child node of html/body/div/div, anywhere.


The yq wrapper (by Andrey Kislyuk) around the JSON processor jq comes with an XML parser wrapper called xq. You can use that too:

xq -x 'del(.. | .div? | select(."@class"? == "portlet solid author-note-portlet"))' file

The -x (--xml-output) option gives you XML output rather than JSON output. Using xq with -i (--in-place) will make it do in-place editing.

This XML parser doesn't care about namespaces.

0
-1

A separate note would be, given that you can achieve the filtering you want with xml_grep, you would have solved the problem in much less time than it would have taken to write up your question to use something like the following bash commands

    mkdir temp
    for file in <subdir>/*.xhtml; do
        # Your magic xml_grep command
        xml_grep -v 'div[@class="portlet solid author-note-portlet"]' "$file" > "temp/$file"
    done
    rm -r subdir
    mv temp subdir

On the flipside there is merit and satisfaction in learning to use other tools.

4
  • Yes, I know I could do that. I mentioned it in my question, where I explicitly stated that I didn't want to do it because it would be slow and tedious (largely because of the overhead of running it once per file, rather than running xmlstarlet once with hundreds of filename args). After setting the namespace, xmlstarlet processed all 350 .xhtml chapter files in 0.196 seconds (on my ancient Phenom II 1090T system)....about the time it took xml_grep to process just one file (0.191 seconds).
    – cas
    Sep 15 at 23:22
  • I feel one of us may be missunderstanding the other: I believe the above commands would process all of your input files in one go. The point is that I would have thought this is then no longer tedious, which I took to be the objection in your post. Although this would quite possibly have a longer runtime, it would be likely much less than writing your original post/getting a cup of tea, etc.
    – eff
    Sep 16 at 9:54
  • Yes, and the script does exactly what I said I didn't want to do - run xml_grep once for each file in a loop. There's significant overhead in starting a perl program 350 times in a loop with one filename per execution versus starting a compiled program once and having it process 350 filename args. It's not that I can't write a trivial script like yours above (especially since it's only a few extra lines in the script I've already written to unzip & re-zip the .epub file containing the .xhtml files), it's that I don't want to, for the reasons I clearly stated in my question.
    – cas
    Sep 17 at 4:33
  • My question was not "how do i write a trivial shell script and have a nice cup of tea?", it was "how do I get xmlstarlet to do what it's supposed to be able to do?". I have received a good answer to my actual question.
    – cas
    Sep 17 at 4:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.