I have the following text file:

#unimportant comment
#possible more unimportant comments
#info1 info2 info3 ,importantname1
#info1 info2 info3 ,importantname2
#info1 info2 info3 ,importantname3

I want to break each file down into separate files. All I really need is to extract the non-commented urls , preserving comments is optional. I want each file to be named such as importantname1.txt or the name following the comma at the end of each comment line appended with .txt

so importantname1.txt would have the following contents:


or possibly

#info1 info2 info3 ,importantname1

so the line would be extracted and saved with the filename after the comment and appended with .txt in this case filename importantname1.txt

In need to do this for each set of lines in the example file. Preserving the comments is unimportant but I need it to be script-able. I also need to account for an unknown number of comment lines in the header. The comment line will always be there before each importanttextX line

  • Will importanttext always be a single line, or are multiple lines possible?
    – FelixJN
    Sep 29 '16 at 9:25


awk -F, '/^#/{f=$NF".txt";cmt=$0; next} {printf "%s\n%s\n",cmt,$0 >f; close(f)}' file


Applied to your sample input:

$ awk -F, '/^#/{f=$NF".txt";cmt=$0; next} {printf "%s\n%s\n",cmt,$0 >f; close(f)}' file

After the above is run, the following files are in the directory:

$ ls
file  importantname1.txt  importantname2.txt  importantname3.txt

The contents of the new files are:

$ cat importantname1.txt 
#info1 info2 info3 ,importantname1
$ cat importantname2.txt 
#info1 info2 info3 ,importantname2
$ cat importantname3.txt 
#info1 info2 info3 ,importantname3

How it works

Awk reads through the input file line by line. Our script classifies those lines as comments or not-comments. For comment lines, the file name and comment is saved. For non-comments, a new file is created and printed

  • `-F,

    This tells awk to use a comma as the field separator on input. In this way, the file name will always be the last field.

  • /^#/{f=$NF".txt";cmt=$0; next}

    If a line begins with #, we save the last field, $NF, plus .txt as a file name f. The whole of the comment line is saved as cmt. We then tell awk to skip the rest of the commands and jump to start over on the next line.

  • printf "%s\n%s\n",cmt,$0 >f; close(f)

    For non-comment lines, we print the last seen comment, cmt, and the current line, $0, into last seen file name f. We then close the file handle for f.

Protecting against bad file names

If the fields that are to be used as file names contain /, the OS will interpret the file names as including directories. To avoid that, we can replace all / with - using gsub(/\//, "-", f) as follows:

awk -F, '/^#/{f=$NF".txt";gsub(/\//, "-", f); cmt=$0; next} {printf "%s\n%s\n",cmt,$0 >f; close(f)}' file
  • Seems to work but I think I am running into an issue with a "/" character which may need to be escaped or otherwise manipulated for a filename. Maybe replacing "/" with "-" ?
    – user178167
    Sep 29 '16 at 6:55
  • @user178167 Yes, slashes would cause problems and your suggestion is excellent. See update to the answer for code to make the substitution.
    – John1024
    Sep 29 '16 at 7:03

A combination of grep and csplit could do the job, by a) grepping all non-commeneted lines plus the preceeding info one and b) splitting the output based on the info comment line:

grep -v -B1 '^#' file | csplit -z - '/^#/' '{*}'

I.e. do not -v extract lines that have a # at the beginning ^# but one line preceeding such lines -B1. Then split the incoming piped input - at each # at the beginning of a line, ignore empty files -z and do so as often as possible {*}.

Renaming would have to be a separate step (csplit autonames the outpit as xx00, xx01 ... - change pre- and suffix with the -f and -b options, repectively)

for f in xx* ; do
   mv "$f" "$( sed -n '2p' "$f" )".txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.