I have a file like this


I need to change this to


I am thinking the code like this

while read line
        sed -i 's/^/{"randomId":{"s":"/' test
        echo $line

        echo $line | grep -q ":"
        [ $? -eq 0 ] && echo "/"{"id":{"s":/"
        [ $? -eq 1 ] && echo "/",{"userId":{"s":/"
        [ $? -eq 2 ] && echo "/",{"host":{"s":/"
        [ $? -eq 3 ] && echo "/",{"status":{"s":/"
        echo "$line | " ";
done < test

Adding for first occurence of : {"id":{"s": and then second occurence adding {"userId":{"s":


Because your data is delimited and easy to read there are a few ways to do this. Sed can parse your data in one line and output the changes:

sed -r -i 's/^(.*):(.*):(.*):(.*):(.*)$/{"randomId":{"s":"\1"},"id":{"s":"\2"},"userId":{"s":"\3"},"dns":{"s":"\4"},"status":{"s":"\5"}}/' input.txt

You use capture groups to capture everything between the beginning of the file, your delimiter, and the end of the file and then just manipulate text around those groups. Every capture group is referenced by "\#" where # is the number of the capture group starting at one and incrementing by one for each group.

As already mentioned you can also set your own delimiter. Bash has a builtin variable called IFS (internal field separator). IFS defaults to whitespace but you have the ability to change it. I won't show the bash example as its already been given and would just be a copy of it.


That grep/echo block isn't going to do anything useful; $? is going to be set once—it's not going to iterate through the fields.

Thankfully, there appears to be a much easier way to do this: just split the fields apart into variables. Thankfully, read can do this for you:

while IFS=':' read -r randomid id userid dns status; do
    printf '{"randomId":{"s":"%s"},"id":{"s":"%s"},"userId":{"s":"%s"},"dns":{"s":"%s"},"status":{"s":"%s"}}\n' \
           "$randomid" "$id" "$userid" "$dns" "$status"

Using printf instead of the more-familiar echo avoids all the \"-sequences echo would require. Do note the backslash at the end of the line to split it.

BTW: The format you're producing is called JSON, and there might be tools to help generate it (for example, jq). Also, it can require its own escaping if, e.g., your fields can contain double-quotes.


With perl:

perl -MJSON -F: -ple '@A = qw/randomId id userId dns status/; $_ = encode_json({map { shift @A => { "s" => $_ } } @F } )' input.csv
  • 1
    This is a nice alternative because it actually uses a JSON generator, so will work even if there is a quote (etc.) in the input field. But you really ought to explain it better, and also note the caveat that it doesn't exactly generate the requested output (fields are in a different order), but that it's equivalent by the rules of JSON. – derobert Mar 24 '17 at 18:11
perl -F: -pale '
   @A = qw/randomId id userId dns status/;
   ($k, $_) = (0, "{" . join(",", map qq/"$A[$k++]":{"s":"$_"}/, @F) . "}");
' yourfile


The @F holds the fields split on : which are then sewn together with appropriate massaging {"s":"fieldI"} and prefixed with the corresponding element from the array @A. All these elements are brought together by a join on , and enclosed in "{" ... "}". And you're done.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.