2

I have a drupal website, which exports its configuration into yaml files. For its webform component, we have some 200 webforms that blind-carbon-copy submissions to an internal email address, to track

We want to remove that email from all of our webforms, and the easiest way to do that is to remove it from the yaml configuration files, and re-import them.

So, my current workflow is to open one of the files, find the line that has email_webdev:, and then delete it and the next 42 lines, save the file, open the next, wash , repeat, for some 200 files.

I'm looking for a one-liner or script that will automatically remove an ordered set of identical 42 lines from all of the files in which they appear.

Technical explanation

In our config directory, there are some 200 files which I want to remove an identical set of lines from:

$ ls webform.webform.*
webform.webform.incoming_student_housing_applica.yml
webform.webform.info_for_students.yml
webform.webform.info_request_for_viewbook.yml
webform.webform.inquire_about_a_project.yml

Each of the webform configuration yaml is a few hundred lines of configuration data, indented, with basic formatting. Heres an example excerpt, with leading line numbers:

....
170   test:
171     roles: {  }
172     users: {  }
173     permissions: {  }
174   configuration:
175     roles: {  }
176     users: {  }
177     permissions: {  }
178 handlers:
179   email_webdev:
180     id: email
181     label: 'Webdev Email'
182     handler_id: webdev_email
183     status: true
184     conditions: {  }
185     weight: 0
186     settings:
... 
214       parameters: {  }
215 variants: {  }
216 uuid: 6073470f-bb3b-40ad-8440-a7cb5f3be4d2

The 42-line stanza, from lines 179-214 in the excerpt above, is what I want to remove from some 200+ files in this directory. So what I'm doing manually, is removing lines 179-214 in vim, and saving. The result looks like this:

...
170   test:
171     roles: {  }
172     users: {  }
173     permissions: {  }
174   configuration:
175     roles: {  }
176     users: {  }
177     permissions: {  }
178 handlers:
179 variants: {  }
180 uuid: 6073470f-bb3b-40ad-8440-a7cb5f3be4d2

It's 42 lines long. Those 42 lines will be identical between files, and in the exact same order, but the stanza starts at different places in the file. For instance, in one file it might begin at line 1068, in another, 872.

$ grep -n email_webdev *
webform.webform.404.yml:183:  email_webdev:
webform.webform.accommodations_letter_request_fo.yml:219:  email_webdev:
webform.webform.agency_survey.yml:219:  email_webdev:
...

All I've been able to figure out so is use the grep switch -A to find the lines after the pattern match:

$ grep -A42  email_webdev *
webform.webform.volunteer_sign_up.yml:  email_webdev:
webform.webform.volunteer_sign_up.yml-    id: email
webform.webform.volunteer_sign_up.yml-    label: 'Webdev Email'
webform.webform.volunteer_sign_up.yml-    handler_id: webdev_email
webform.webform.volunteer_sign_up.yml-    status: true
webform.webform.volunteer_sign_up.yml-    conditions: {  }
webform.webform.volunteer_sign_up.yml-    weight: 0
webform.webform.volunteer_sign_up.yml-    settings:
...

So, those are the lines that I want to remove from that file (and all the files in the directory), But, I can't just have it remove the line id: email, because that appears in other email respones, nor can I just pattern match any other line, like weight: 0, which appears in almost every other element, too, and conditions: { }. They can only the the lines in this stanza, which begins with email_webdev:, and continues on through the next 42 lines, identically, in every file.

Is there an easier way to do this?

3
  • May you provide a minimal, reproducible example? I am a bit confused.
    – Quasímodo
    Jun 25, 2020 at 13:57
  • @Quasímodo I apologize for being unclear. I've tried to better summarize the background and my ask at the top, and then expounded on technical details in the body. Let me know if this is helpful or not.
    – user394
    Jun 25, 2020 at 14:33
  • Yes, it is clear now, thanks!
    – Quasímodo
    Jun 25, 2020 at 14:46

2 Answers 2

4

Please try this without the mv first to check if the new .out files created are exactly as you want. Re-enable the mv operation to overwrite the original files.

for file in webform.webform.*; do
    awk '/^  email_webdev:$/{n=42}{if(!n){print}else{n--}}' "$file" > "$file.out"
    mv -- "$file.out" "$file"
done

The awk script detects a line exactly like this:

  email_webdev:

I.e., two spaces, email_webdev:, end-of-line character.

Once that line is detected, it sets n=42, thus stopping printing the lines, and then only starts printing again after 42 lines have been counted.

1

You can use the ex line editor to do this.

for f in webform.webform.*; do
  printf '%s\n' /email_webdev:/ .,+41d x | ex -s "$f"
end

Working:

  • Whenever we see relative addressing, we should reach for the ed/ex line editors, unless the file sizes are huge, because they support a wide variety of line addressing modes.
  • printf is emitting ex editor code, which ex reads on its stdin and applies on the input file.
  • First we make the email_webdev as the current line by searching for it: /email_webdev:/.
  • Then, starting from the current line, we grab the 42 chunk of lines .,.+41d.
  • The deed is done, so now we save and quit: x

PS: The 179-214 range is not 42 lines.

2
  • +1 This is actually a very cool solution. Would you please explain what ex is doing there? I'm sure few users are familiar with it.
    – Quasímodo
    Jun 25, 2020 at 21:40
  • Thank you for the kind words. I have given a brief working method. Jun 25, 2020 at 22:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .