This questions has been asked before, except they were looking for text between two dates including or excluding those dates. I’ve changed one of the answers so that it selects and includes from first date and goes to but excludes the second date (I think/hope). However I don't understand where you provide the text file in which to apply this to?

set - date1 date2 "junk"  


# Output lines between two parameter dates 
# INCLUDING the first parameter date but not the second

awk -v from=$from -v till=$till '
    ($2 >= from) && ($2 < till) { print $0 ; next }
    ($2 >= till) { exit }' "$file"

Since discovering a clumpy solution, I've realized that there's another aspect that would improve this, and prevent me from having to write slight changes of the script line many times over.

Current working solution is:

awk '/^date_1_/,/^date_2_/ {print}' file.txt > file2.txt
grep -v "date_2_" file2.txt > file2tmp.txt
mv file2tmp.txt file2.txt

However I would like it to do this for every new date. e.g. select text from Date_1_ (including date_1_) up to but not including Date_2_, then select text from Date_2_ to 3 in the same way, 3 to 4 in the same way (all the way to 1000). Is there a solution I can easily scale up?

Example input text file (though actual file goes up to 1000, also there are not spaces between the underscores in the txt file but i dont have a back tick to escape there italic effects):

' > _ 1_ fe fi fo fum >_ 2_ beep bap bop >_ 3_ ti fi at at

Example Output:
created text file1: > _ 1_ fe fi fo fum
created text file2: >_ 2_ beep bap bop
created text file3: >_ 3_ ti fi at at

  • What's wrong with sed -e '/string1/,/string2/-1p ? – waltinator Jul 11 '16 at 13:38
  • This doesn't seem to work for me, not sure why. this is what i wrote when using this: sed -e '/date1/,/date2/-1p file.txt > filedate1_2.txt – Giles Jul 11 '16 at 13:53
  • Where are you planning to source the dates from? argv[]? – Alex Stragies Jul 11 '16 at 19:43
  • sorry, i used dates as an example, as it was a bit of a branch from someone elses question. but for me it doesn't need to be actual dates, but each section is marked by _ 1_ , _ 2_ , _ 3_ etc – Giles Jul 11 '16 at 20:03

If I understand the current state of this constantly moving question correctly, you have a file with an arbitrary number of lines, divided into sections marked by something like _1_ to _1000_, and you would like to split those sections into individual files. If so, then csplit can do that:

 csplit file.txt '/^_[0-9]\+_/' '{*}'
| improve this answer | |
  • worked perfectly, thank you, i was really hoping there would be something more simple like this, csplit is brilliant! – Giles Jul 12 '16 at 10:17

This awk program will accomplish what you wanted to do at first in one command:

awk '/^date_1_/,/^date_2_/ { if (prev) print prev ; prev=$0 }' file.txt > file2.txt

You tested this, and it worked. Then you extended the question to be able to operate on a range of generic row markers given as pairs of arguments.

You can create a comfortable workflow for this by defining 2 helper functions:

  • One creates the awk program to feed in through process substitution
  • The other runs awk with the parameters fed through to the first helper:


fun1(){ while [ ! -z $1 ] && [ ! -z $2 ] ; do echo "
/^$1/,/^$2/ {if (\$0~/^$1/) prev=\"\";
if (prev) print prev; prev=\$0 }"; shift; shift ; done }

fun2(){ awk -f <(fun1 $@); }

# Example data, example ranges, but could be any string:
seq 1 13 | sed -e 's/.*/_&_/' | fun2 _2 _4 _9 _11

Solution for Version 3 of the Problem:

fun3(){ echo "/^$1/,/^$2/ {if (\$0~/^$1/) prev=\"\";
if (prev) print prev; prev=\$0 }"; }

fun4(){ ifile=$1; shift; while [ ! -z $1 ] && [ ! -z $2 ] ; do
        awk -f <(fun3 $1 $2) $ifile > $1.txt ; shift; done }

# Create example data file:
seq 1 13 | sed -e 's/.*/_&_/' > inputData.txt

fun4 inputData.txt _2 _5 _8 _12

ls _*
_2.txt  _5.txt  _8.txt
| improve this answer | |
  • This works well and is far neater than my solution, I've adapted the question since finding my clumpier solution - do you know how i could adapt this to do what its currently doing between date_1_ and date_2_ but to have it do the same between 2 and 3, 4 and 5, 5 and 6 etc up to 1000? – Giles Jul 11 '16 at 20:19
  • feel like I just entered the deep end of the pool without my arm bands. I understand the principles you explained, and i understand parts of this code but I can't get my head around exactly how it works. would this be able to increase in incremental amounts? i.e. get the text between 1 and 2 and output to file 1, text between 2 and 3 and output to file 2, and so on so forth for 3-4 4-5 5-6 6-7, in this pattern until reaching 1000 – Giles Jul 11 '16 at 20:57
  • so using your method I could set the sequence of numbers to 1000 using seq 1 1000, pipe into sed (which i don't quite understand its application here - is it to state an undefined amount of text between each _ number_ (_& _)) and then pipe that into which numbers to run it on within the given range? and if so where do i tell it which file to use? and can i use the usual output method > ? – Giles Jul 11 '16 at 21:36
  • sorry my page hadn't up dated so my comment is out of sink with your last adjustment, i think i'm getting it now. going to play around with it and see how things go – Giles Jul 11 '16 at 21:46
  • s/.*/_&_/ is the same as, but shorter than s/^\(.*\)$/_\1_/. I used this with seq to generate sample data, that would have the same kind of row markers as your data. You said "each section is marked by _ 1_ , _ 2_ , _ 3_ etc". Perhaps put some sample data in your q? – Alex Stragies Jul 11 '16 at 21:54

I did some more reading around and pieced some things together, this now works for me.

awk '/^date_1_/,/^date_2_/ {print}' file.txt > file2.txt
grep -v "date_2_" file2.txt > file2tmp.txt
mv file2tmp.txt file2.txt

But it still requires 2 steps and a tempfile.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.