0

I have searched and I don't know what I'm doing wrong but I can't find an answer to this question.

I have a file with all text stored as a single line. I am needing to find a pattern and remove all the text before and after that text until delimiters.

Ex. File

[{"something":false,"more":"123","moresamerecord":"otherstuff"},{"something":false,"more":"abc","moresamerecord":"otherstuff"},{"something2":false,"more":"def","moresamerecord":"otherstuff"},{"something2":false,"more":"456","moresamerecord":"otherstuff"}]

Keep in mind this is a single line with multiple records. I'm trying to find "abc" and remove everything between the previous and next record.

Expected outcome should be this.

[{"something":false,"more":"123","moresamerecord":"otherstuff"},{"something2":false,"more":"def","moresamerecord":"otherstuff"},{"something2":false,"more":"456","moresamerecord":"otherstuff"}]

I have been trying and unable to figure this out, any help would be appreciated.

5
  • 3
    This looks like JSON code. You are better off with a tool that specializes in parsing this, such as jq. However, first you would have to ensure that the line has correct JSON syntax (the current example ends with a comma, which violates the syntax). Feb 8, 2021 at 2:03
  • 1
    Yes, you are correct the actual file does not end in a comma, that's my fault in typing out the example.
    – ChSpan
    Feb 8, 2021 at 2:31
  • I have to prove that I once learned computer science. sed and awk are good at working with regular expressions, i.e. expressions that can be described with a regular or Chomsky-3 grammar. JSON, however, is not regular but context-free or Chomsky-2, if I am not totally wrong. Thus, these tools are not well suited to parsing JSON. Feb 8, 2021 at 2:37
  • 1
    so what if that abc was in somewhere else in some records like in {"something_abc":false,"more":"not_a_b_c","moresamerecord":"otherstuff"}? where that abc should be in order to remove that record? Feb 8, 2021 at 7:41
  • Please add the jq category for your question as it really is one of jq.
    – guest_7
    Feb 8, 2021 at 8:47

4 Answers 4

1

As has been noted, jq is the tool to use for this type of data. However, jq does impose certain syntactical constraints, such as "lists of objects need to be in an array denoted by square brackets".

If you aren't able to make sure the file is already valid json, you can use sed to preprocess it (and we'll do an initial run through jq because the result is easier on the eye, while also being checked for correctness.)

$ sed 's/^/[/; s/,$/]/' data.txt | jq -r '.[]'
{
  "something": false,
  "more": "123",
  "moresamerecord": "otherstuff"
}
{
  "something": false,
  "more": "abc",
  "moresamerecord": "otherstuff"
}
{
  "something2": false,
  "more": "def",
  "moresamerecord": "otherstuff"
}
{
  "something2": false,
  "more": "456",
  "moresamerecord": "otherstuff"
}

Now, let's modify the jq command to remove any object matching "more": "abc":

$ sed 's/^/[/; s/,$/]/' data.txt | jq -r '.[] | select(.more != "abc")'
{
  "something": false,
  "more": "123",
  "moresamerecord": "otherstuff"
}
{
  "something2": false,
  "more": "def",
  "moresamerecord": "otherstuff"
}
{
  "something2": false,
  "more": "456",
  "moresamerecord": "otherstuff"
}

Finally, it seems like you also want a post-processing step to squash it back into one line with comma-delimiters and no whitespace:

$ sed 's/^/[/; s/,$/]/' data.txt | jq -r '.[] | select(.more != "abc")' | sed 's/}$/},/' | tr -d ' \n'
{"something":false,"more":"123","moresamerecord":"otherstuff"},{"something2":false,"more":"def","moresamerecord":"otherstuff"},{"something2":false,"more":"456","moresamerecord":"otherstuff"},
1
  • 1
    I really appreciate everyones help, in the end I did use jq as it was the most effective way of doing what I needed. I modified the code a little to fit my needs but this was the best method thank you.
    – ChSpan
    Feb 8, 2021 at 15:33
0

The basic idea is to extend the pattern to the delimiters and not any further.

So to match from the nearest { to the "abc" you can look for a { followed by any characters which are not {. Likewise you can extend from "abc" to the nearest following } by looking for characters which are not } followed by `}'.

Then there are a couple of edge cases to handle the commas.

sed 's/{[^{]*"abc"[^}]*}//;s/,,/,;s/,$//;s/^,//'

If you data is any more complicated than you show, and in particular if the { and } can get nested then you probably want to switch to parsing. Regular expressions "can't count" so whilst you can write patterns that handle any particular finite depth (e.g. 3) you can't handle arbitrary depths.

The advice in the comments to use jq is certainly worth trying, rather than using sed.

1
  • I am definitely looking into jq to see if that is a viable solution.
    – ChSpan
    Feb 8, 2021 at 3:01
0

If jq is not a solution, I propose this:

# Instead of a single line pattern matching,
# make the "records" one per line
# then delete the line with the pattern
# finally get everything again to a single line
sed -e 's:,{:\n{:g;s:,$::' file | sed '/abc/d' | tr '\n' ','

Step by step:

$ sed -e 's:,{:\n{:g;s:,$::' file
{"something":false,"more":"123","moresamerecord":"otherstuff"}
{"something":false,"more":"abc","moresamerecord":"otherstuff"}
{"something2":false,"more":"def","moresamerecord":"otherstuff"}
{"something2":false,"more":"456","moresamerecord":"otherstuff"}
$ sed -e 's:,{:\n{:g;s:,$::' foo.txt | sed '/abc/d'
{"something":false,"more":"123","moresamerecord":"otherstuff"}
{"something2":false,"more":"def","moresamerecord":"otherstuff"}
{"something2":false,"more":"456","moresamerecord":"otherstuff"}
$ sed -e 's:,{:\n{:g;s:,$::' foo.txt | sed '/abc/d' | tr '\n' ','
{"something":false,"more":"123","moresamerecord":"otherstuff"},{"something2":false,"more":"def","moresamerecord":"otherstuff"},{"something2":false,"more":"456","moresamerecord":"otherstuff"},
0
awk '
  BEGIN { FS = "},{" }
  { k=0
    for (i=1; i<=NF; i++)
      if ($i !~ /"abc"/)
        printf "%s%s", (k++?FS:""), $i
    $0=""
  }1
' file

$ cat file \
| sed -e 's/},{/}\n{/g'           \
| sed -E '/([{:,])"abc"([,:}])/d' \
| paste -sd, -                    \
;
  • Separate out the records to one / line.
  • Now remove any record containing "abc"
  • Stitch back the records with a comma ,

Output:

{"something":false,"more":"123","moresamerecord":"otherstuff"},{"something2":false,"more":"def","moresamerecord":"otherstuff"},{"something2":false,"more":"456","moresamerecord":"otherstuff"}

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .