0

I want to use sed to search for line breaks that are not preceded by }. The regex for identifying theses cases is: [^}]$ . What I want to say: If the line ended without } then remove the new line. Any line should end with } before the newline.

How to put that in sed to remove them?

EDIT:

Example:

{'Date': 'Fri, 19 Apr 2019 07:23:14 GMT', 'Server': 'Apache', 'Vary': 'Qualys-Scan', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains;preload', 'Set-Cookie': 'ASP.NET_SessionId=ivoa5bhet0s2ygkylmimvkie; path=/; secure; HttpOnly;SameSite=strict, SC_ANALYTICS_GLOBAL_COOKIE=12f133ea5080403692b4ce458fd1a540; expires=Thu, 19-Apr-2029 07:23:14 GMT; path=/; secure; HttpOnly;SameSite=strict, SC_ANALYTICS_SESSION_COOKIE=336B597E7A534D6393C57DF11E047484|1|ivoa5bhet0s2ygkylmimvkie; path=/; secure; HttpOnly;SameSite=strict, incap_ses_885_270026=cDp/VlO1AHgshF9F6SZIDGJ3uVwAAAAAg7DwpecyehBCyhXgoYO5GA==; path=/; Domain=.zurich.co.uk, ___utmvmykuNyVY=dlNaoEsuXSO; path=/; Max-Age=900, ___utmvaykuNyVY=nWJx01KvGT; path=/; Max-Age=900, ___utmvbykuNyVY=JZy
    XEtOwalQ: PtR; path=/; Max-Age=900', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'private', 'Content-Type': 'text/html; charset=utf-8', 'Keep-Alive': 'timeout=5, max=10', 'Connection': 'Keep-Alive', 'X-Iinfo': '8-3925806-3925807 NNNN CT(73 151 0) RT(1555658593583 5) q(0 0 3 0) r(6 6) U5', 'X-CDN': 'Incapsula', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked'}

When I applied your script, it combined the lines. However, the new line starts with tab. How to also remove the spaces if the newline starts with spaces?

  • So if a line does not end with }, you want to join the next line to it? – steeldriver May 11 at 12:59
  • yes. just remove it. – None May 11 at 13:00
0

You can use the following to join lines where the preceding line does not end with }:

sed -e :a -e '/}$/!N; s/\n//; ta'

This is essentially a variant of 39. Append a line to the next if it ends with a backslash "\" from Sed One-Liners Explained, Part I: File Spacing, Numbering and Text Conversion and Substitution, replacing the "ends with a backslash" condition with "does not end with a right brace".

Ex. given

$ cat > input
foo bar
baz}
foo bar baz}

then

$ sed -e :a -e '/}$/!N; s/\n//; ta' input
foo barbaz}
foo bar baz}

If you want to also remove any leading horizontal whitespace at the start of the joined line, you can change s/\n// to s/\n[[:blank:]]*//

Testing with your example (after converting the leading spaces to tabs):

$ cat example
        {'Date': 'Fri, 19 Apr 2019 07:23:14 GMT', 'Server': 'Apache', 'Vary': 'Qualys-Scan', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains;preload', 'Set-Cookie': 'ASP.NET_SessionId=ivoa5bhet0s2ygkylmimvkie; path=/; secure; HttpOnly;SameSite=strict, SC_ANALYTICS_GLOBAL_COOKIE=12f133ea5080403692b4ce458fd1a540; expires=Thu, 19-Apr-2029 07:23:14 GMT; path=/; secure; HttpOnly;SameSite=strict, SC_ANALYTICS_SESSION_COOKIE=336B597E7A534D6393C57DF11E047484|1|ivoa5bhet0s2ygkylmimvkie; path=/; secure; HttpOnly;SameSite=strict, incap_ses_885_270026=cDp/VlO1AHgshF9F6SZIDGJ3uVwAAAAAg7DwpecyehBCyhXgoYO5GA==; path=/; Domain=.zurich.co.uk, ___utmvmykuNyVY=dlNaoEsuXSO; path=/; Max-Age=900, ___utmvaykuNyVY=nWJx01KvGT; path=/; Max-Age=900, ___utmvbykuNyVY=JZy
        XEtOwalQ: PtR; path=/; Max-Age=900', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'private', 'Content-Type': 'text/html; charset=utf-8', 'Keep-Alive': 'timeout=5, max=10', 'Connection': 'Keep-Alive', 'X-Iinfo': '8-3925806-3925807 NNNN CT(73 151 0) RT(1555658593583 5) q(0 0 3 0) r(6 6) U5', 'X-CDN': 'Incapsula', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked'}

$ sed -e :a -e '/}$/!N; s/\n[[:blank:]]*//; ta' example
        {'Date': 'Fri, 19 Apr 2019 07:23:14 GMT', 'Server': 'Apache', 'Vary': 'Qualys-Scan', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains;preload', 'Set-Cookie': 'ASP.NET_SessionId=ivoa5bhet0s2ygkylmimvkie; path=/; secure; HttpOnly;SameSite=strict, SC_ANALYTICS_GLOBAL_COOKIE=12f133ea5080403692b4ce458fd1a540; expires=Thu, 19-Apr-2029 07:23:14 GMT; path=/; secure; HttpOnly;SameSite=strict, SC_ANALYTICS_SESSION_COOKIE=336B597E7A534D6393C57DF11E047484|1|ivoa5bhet0s2ygkylmimvkie; path=/; secure; HttpOnly;SameSite=strict, incap_ses_885_270026=cDp/VlO1AHgshF9F6SZIDGJ3uVwAAAAAg7DwpecyehBCyhXgoYO5GA==; path=/; Domain=.zurich.co.uk, ___utmvmykuNyVY=dlNaoEsuXSO; path=/; Max-Age=900, ___utmvaykuNyVY=nWJx01KvGT; path=/; Max-Age=900, ___utmvbykuNyVY=JZyXEtOwalQ: PtR; path=/; Max-Age=900', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'private', 'Content-Type': 'text/html; charset=utf-8', 'Keep-Alive': 'timeout=5, max=10', 'Connection': 'Keep-Alive', 'X-Iinfo': '8-3925806-3925807 NNNN CT(73 151 0) RT(1555658593583 5) q(0 0 3 0) r(6 6) U5', 'X-CDN': 'Incapsula', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked'}
  • Can you explain the command with details? I do not get most of it. What does :a is for? and the rest etc. It is complex – None May 11 at 13:18
  • @None did you read the explanation in the provided link? I don't think I can do better than that – steeldriver May 11 at 13:23
  • when I used your script, the first line that ends with utmvbykuNyVY=JZy converted to: `utmvaykuNyVY=nW` (the JZy replaced with nW\. Also the empty spaces at the beginning of the new line needs to be removed. Sorry I am not very familiar with Linux. – None May 11 at 13:39
  • @None you can scoop up arbitrary horizontal whitespace by changing s/\n// to s/\n[[:blank:]]*// – steeldriver May 11 at 13:45
  • Your command does not work with me. Can you plz check my edit in the post? – None May 11 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.