0

I have a HLS playlist file looking like this:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:12
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:12.500000,
playlist0.ts
#EXTINF:8.333333,
playlist1.ts
#EXTINF:12.500000,
playlist2.ts
....

Then I have a file with links (signurls.txt) looking like this:

https://example.com/playlist0.ts?Sign=xyz&Exp=1639139375&AWSAccessKeyId=abc
https://example.com/playlist1.ts?Sign=yzx&Exp=1639139375&AWSAccessKeyId=bca
https://example.com/playlist10.ts?Sign=zyx&Exp=1639139375&AWSAccessKeyId=cab
....

I am trying to insert the links into the .m3u8 file like this:

....
#EXTINF:12.500000,
https://example.com/playlist0.ts?Sign=xyz&Exp=1639139375&AWSAccessKeyId=abc
....

I have come up with this piece of script:

for f in *.ts; do 
   sed -i '' -e "'s|$f|`grep -e $f signurls.txt`|'" playlist.m3u8;
done

EDIT: Each of the .ts files listed in the playlist.m3u8 file is present in the current directory. I used the actual files for the for loop so that the playlist file can be changed without issues.

The reason for the -i '' is that it needs to work both on macOS and Linux.

I tried to echo the sed string and I can see that the string expansion is working as expected.

However, when I run the script I get this error (script in one line):

sed: 1: "'s|playlist0.ts|https:/ ...": invalid command code '
2
  • Please add your desired output (no description, no images, no links) for that sample input to your question (no comment).
    – Cyrus
    Commented Dec 10, 2021 at 22:50
  • @Cyrus The desired output is listed in the snippet following "I am trying to insert the links into the .m3u8 file like this:" - it's abbreviated because it is highly repetitive.
    – hstr
    Commented Dec 11, 2021 at 13:01

4 Answers 4

3

I'm going to ignore the fact that you seem to have files called something.ts which your code is apparently using. You do not mention these in the text, so I will pretend I don't know about them.

$ cat urls
https://example.com/playlist0.ts?Sign=xyz&Exp=1639139375&AWSAccessKeyId=abc
https://example.com/playlist1.ts?Sign=yzx&Exp=1639139375&AWSAccessKeyId=bca
https://example.com/playlist10.ts?Sign=zyx&Exp=1639139375&AWSAccessKeyId=cab
$ cat playlist.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:12
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:12.500000,
playlist0.ts
#EXTINF:8.333333,
playlist1.ts
#EXTINF:12.500000,
playlist2.ts
$ awk -F'[/?]' 'NR==FNR { pl[$4]=$0; next } /^[^#]/ && ($0 in pl) { $0 = pl[$0] }; 1' urls playlist.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:12
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:12.500000,
https://example.com/playlist0.ts?Sign=xyz&Exp=1639139375&AWSAccessKeyId=abc
#EXTINF:8.333333,
https://example.com/playlist1.ts?Sign=yzx&Exp=1639139375&AWSAccessKeyId=bca
#EXTINF:12.500000,
playlist2.ts

The awk command above first reads the URLs from the first file (urls) and adds the .ts strings as keys in the an associative array called pl. The complete URL is added as the values of the array.

The .ts strings are found by treating each URL as a slash-or-question-mark-delimited string, and picking out the fourth field from that string.

The code then reads lines from the .m3u8 file, and for each line that does not start with a # character, it tests whether the line is a key in the pl array. If it is, the current line is replaced with the corresponding URL from the array. All lines of the .m3u8 file (possibly modified as just described) are then printed.

In the example above, you can see that the last entry is not replace in the playlist file, as the URL for that entry is not in the urls file.

2

Don't use a shell while or for loop to process text. See Why is using a shell loop to process text considered bad practice? for reasons why.

Instead, use perl or awk or python or any-language-that-isn't-shell for text processing.

Here's a perl one-liner which uses the Getopt::Std module (a core library module, included with perl), allowing the signurls file can be specified with -s, so that it can be read in and processed separately from the playlist file(s) - this is important because we don't want that file to be modified by the -i option.

This could have been done with something like $signurls = shift; (first arg) or $signurls = pop; (last arg) but a) that would mean the signurls file had to be the very first (or last) argument (which is inflexible, but not necessarily a bad thing for a quick-and-dirty hack), b) it would have been more complicated and less reliable to have a default filename for $signurls, and c) it's really no harder to do it with Getopt::Std, and that's a useful library module to know how to use.

All subsequent arguments are treated as playlist file(s). Because they are processed with a while(<>), they can be modified in-place by perl's -i option:

$ perl -MGetopt::Std -i.bak -lpe '
  BEGIN {
    # Parse any command line options.
    getopts("s:", \%opts);
    my $signurls = $opts{s} // "signurls.txt";

    # Read in signurls file and build hash containing patterns
    # and replacement strings.
    open($fh,"<",$signurls) || die "error opening \"$signurls\": $!\n";
    while(<$fh>) {
      chomp;
      # Extract the "filename" portion of the URL and use it as the hash's key.
      # the hash's value is the URL itself.
      m=^.*://.*?/([^/]*)[/?].*=;
      $urls{$1} = $_;
    };
    close($fh);
  };

  foreach my $f (keys %urls) {
    if ($_ eq $f) {
      $_ = $urls{$f};
      last;   # we already matched, so there's no need to
              # compare this line against the remaining keys.
    };
  };' -s signurls.txt playlist.m3u38

Note 1: the .bak after -i causes perl to make a backup copy (with .bak extension) of each original input (playlist) filename.

Note 2: the BEGIN { ... } code block is executed once before any files are opened or processed. The remainder of the script, outside of the BEGIN block, is executed once for each and every line of data from the input file(s).

Sample output after run:

$ cat playlist.m3u8 
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:12
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:12.500000,
https://example.com/playlist0.ts?Sign=xyz&Exp=1639139375&AWSAccessKeyId=abc
#EXTINF:8.333333,
https://example.com/playlist1.ts?Sign=yzx&Exp=1639139375&AWSAccessKeyId=bca
#EXTINF:12.500000,
playlist2.ts

I could have written this so that it had a -P option for the playlist file. In fact, I first started writing it that way. But by writing it to use perl's -p option (which is not much more than an automatic while(<>) loop - see man perlrun) to read and process the playlists, I could use perl's -i option and have the script edit the playlist file(s) in-place without having to write my own in-place editing code. It also added support for processing multiple input files without any additional code required. Two useful features, for free.

1

Using sed in a loop

$  while read line; do sed -i.bak "s#$(sed 's#.*/\([^?]*\).*#\1#' <<< $line)#$line#" playlist.m3u8; done < signurls.txt

$ cat playlist.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:12
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:12.500000,
https://example.com/playlist0.ts?Sign=xyzplaylist0.tsExp=1639139375playlist0.tsAWSAccessKeyId=abc
#EXTINF:8.333333,
https://example.com/playlist1.ts?Sign=yzxplaylist1.tsExp=1639139375playlist1.tsAWSAccessKeyId=bca
#EXTINF:12.500000,
playlist2.ts
....
1

The error message given was actually better than I thought, it turns out that it was indeed the ' character that was the problem.

Removing the ' characters in the sed pattern string solved the problem:

for f in *.ts; do                                                                                                                            
    sed -i "" "s|$f|$(grep -e $f signurls.txt)|g" playlist.m3u8 ;                                                                            
done

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .