2

vtt files look like this:

WEBVTT

1
00:00:00.096 --> 00:00:05.047
you're the four functions if you would of 
management first of all you have the planning

2
00:00:06.002 --> 00:00:10.079
the planning stages basically you were choosing appropriate 
 organizational goals and courses

3
00:00:11.018 --> 00:00:13.003
action to best achieve those goals

I need just the text, like this:

you're the four functions if you would of management first of all you have the planning the planning stages basically you were choosing appropriate organizational goals and courses action to best achieve those goals

on ubuntu I tried:

cat file.vtt | grep -v [0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9][0-9][[:space:]][[:punct:]][[:punct:]][[:punct:]][[:space:]][0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9][0-9]

that gives me:

WEBVTT

1
you're the four functions if you would of 
management first of all you have the planning

2
the planning stages basically you were choosing appropriate 
 organizational goals and courses

3
action to best achieve those goals

but I can't figure out how to do the rest. what I want to replace is

\n[0-9]+\n\n with space but I can't figure out how to make sed or grep do that.

how do I get with basic / portable (eg generally preinstalled in ubuntu, centos, etc, eg grep, sed, or tr command) to just the raw text with the subtitle timing removed, and all in one line (no newlines)?

NOTE: this has to work for other language characters like chinese hindi arabic, so preferably no [a-z] type matches but instead remove the timing lines which are very consistent in format. Also don't blindly remove any numbers as text can contain numbers

NOTE 2: ultimate goal is to have the text safe for a json value , so all special chars removed and double quotes escaped, but that's sort of beyond the scope of this question

  • Or use awk (or perl) in paragraph mode, and simply skip the first couple of fields e.g. awk -vRS= -vORS= -F'\n' '{for (i=3;i<=NF;i++) print $i}' file.vtt – steeldriver Oct 25 '16 at 22:46
  • @don_crissti @steeldriver both of those some how mess up my terminal. I run it and it shows part of one line as though I had just typed it it. eg it shows user@ubuntu$ were anning it's so weird. I reopened my terminal. other commands are fine though. – AwokeKnowing Oct 25 '16 at 23:06
  • 1
    ... possibly your file has DOS line endings (CR-LF)? – steeldriver Oct 25 '16 at 23:07
  • @steeldriver it's certainly possible (and out of my control), how would I account for that? Even then, the text shown seems to be garbled (eg not a whole line but a couple parts of 2 lines) – AwokeKnowing Oct 25 '16 at 23:08
  • There are a whole host of ways (dos2unix, vi's set ff=unix, etc. etc). Probably I'd use sed e.g. sed 's/\r$//' file.vtt | awk -vRS= -vORS= -F'\n' '{for (i=3;i<=NF;i++) print $i} END {print "\n"}' (the final print statement is only to make it look nice in the terminal - you don't need it if you're capturing the text to a variable for example). – steeldriver Oct 25 '16 at 23:16
2

Since your file appears to consist of a sequence of records separated by one or more blank lines, I'd suggest trying something based on the paragraph modes of either awk or perl.

For example, if you always need to strip off the first two lines, like

1
00:00:00.096 --> 00:00:05.047

you could split into newline-delimited fields within blank-separated paragraphs and skip the first two fields using either

awk -vRS= -vORS= -F'\n' '{for(j=3;j<=NF;j++) print $j; print " "}' file.vtt

or

perl -F'\n' -00ne 'print join("", @F[2..$#F]), " "' file.vtt

If you can't rely on there being a fixed number of fields (lines) to be removed, then it's fairly easy to add a regular expression test - a little easier in perl since it allows us to grep directly on arrays rather than writing an explicit loop. For example, to split into blank-separated records and then print only those fields (lines) having at least one sequence of at least 3 alphabetic characters, you could use

perl -F'\n' -00ane '
  print join("", grep { /[[:alpha:]]{3}/ } @F), " "
' file.vtt

If you want to exclude the WEBVTT string you can simply skip the first record, i.e.

perl -F'\n' -00ane '
  print join("", grep { /[[:alpha:]]{3}/ } @F), " " if $. > 1
  ' file.vtt

It will be down to you to choose a suitable regex that capture the wanted lines and excludes the unwanted ones. You can add an END block in either awk or perl if you want to add a final newline to the concatenated output.


NOTE: since (based on the discussion in comments) your files appear to have DOS-style CRLF line endings, you will need to deal with those - either by modifying the field and record separators in the above commands accordingly, or by stripping out the CRs first e.g.

sed 's/\r$//' file.vtt | 
  perl -F'\n' -00ane '
    print join("", grep { /[[:alpha:]]{3}/ } @F), " " if $. > 1
  '
you're the four functions if you would of management first of all you have the planning the planning stages basically you were choosing appropriate  organizational goals and courses action to best achieve those goals steeldriver@xenial-vm:~/test/$
  • Thank you. I hadn't realized perl was included in practically every common distro of linux that would be used on a webserver / docker image. This is a good solution. Is there any standard commandline tool to make the string json safe (remove tabs and escape backstlashes and double quotes) – AwokeKnowing Oct 26 '16 at 18:04
0

ok here's what I ended up with

#!/bin/bash
fname=$1
sed 's/\r$//' "$fname"    |\
grep -v -- "-->"          |\
grep -v "^$"              |\
grep -E -v "^[0-9]+$"     |\
sed 's/WEBVTT//'          |\
tr '\n' ' '               |\
tr -s ' '                 |\
tr -d '\t'                |\
sed 's/\\/\\\\/g'         |\
sed 's/"/\\"/g' 
  1. fix windows newlines
  2. find all lines that don't have -->
  3. find all lines that are not empty (I think this is faster, maybe not)
  4. find all lines that aren't just a number
  5. remove WEBVTT header
  6. remove newlines
  7. squeeze multiple spaces to 1
  8. remove tabs
  9. escape any backslashes (for json)
  10. escape any doublequotes (for json)

Thanks to @steeldriver for the fix for windows new lines.

I wouldn't use this in production as it's a bit weak eg it would skip lines of text like "you are --> my friend" and probably a few other cases, but it should be good enough for my purposes (posting to solr for searching)

I realize it's quite inefficient though. I would appreciate tips on that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.