0

what I'm doing here is a bit complex, so hopefully I can explain it well enough so that it makes sense. The tldr version is that I want to read 10 lines of a file and then run a script based on the input from that file. Once it is done, move on to the next 10 until the file is complete, then end.

Here's the long story. I am trying to do some archives of Usenet newsgroups. I have a list of 42227 news groups that I want to archive. I am using the sinntp application to pull information from the usenet server. It is written in Python and it has a serious bug. If a newsgroup message is missing, it will end the command unsuccessfully without retrying.

My workaround is downloader.sh. If the sinntp fails unsuccessfully, downloader.sh will force it to retry until it completes. Once it completes, it renames and zips up the output file for permanent storage (on archive.org eventually).

downloader.sh:

#!/bin/bash

while getopts g: flag
do
    case "${flag}" in
        g) group=${OPTARG};;
    esac
done

downloader() {
nntp-pull $group
}

until downloader; do
    echo "restarting download" >&2
    sleep 1
done

if [ $? -eq 0 ]; then
    mv -v $group $group.mbox && zip -rm $group.mbox.zip $group.mbox
fi

This will only download one newsgroup at a time. I also wrote a second script, archiver.sh that will read the master newsgroup file and will run downloader.sh in parallel.

archiver.sh

#!/bin/bash

cat newsgroup_list_working.txt | parallel -j 10 ./downloader.sh -g {}

However, it doesn't really work. Too often, it will just stop downloading anything and no new downloads will start. I am not running out of disk or memory and my CPU isn't hitting 100%. Also, my disk i/o isn't maxed out because I can run other things with no problem. The download speed is barely above 1MB so I don't think my network is doing the problem.

Any suggestions would be greatly appreciated!

1
  • It will be helpful if you post 30 lines of the input and show us which 3 commands should run based on the input.
    – Ole Tange
    Dec 25, 2022 at 10:13

1 Answer 1

0

You probably want to read 10 args per command:

cat file | parallel -N10 ./downloader.sh -g {5}

where 5 is the line containing the name of newsgroup.

Use --dry-run to have GNU Parallel show the commands that it would be run:

cat file | parallel --dry-run -N10 ./downloader.sh -g {5}

When that shows the right commands, you can spice it up with -j 300% to run 3 jobs per CPU thread in parallel.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .