Trying to create a while loop to output content of one file to another

Question

I am trying to create a while loop so that it takes content from one file and creates some content on another file. But what i noticed is that it is only creating the last line of the file instead of all the lines in the file. What am i missing here? Or is my approach with echo wrong?

My file called "test" contains a list of strings for ex.

unix_idx
web_pn_iis
wis_healthpartners

I am using the following command to try and create a while loop.

while read -r line;
do
echo "  {
    \"name\": \"$line\",  
    \"datatype\": \"event\", 
    \"searchableDays\": 180, 
    \"maxDataSizeMB\": 0, 
    \"totalEventCount\": \"0\", 
    \"totalRawSizeMB\": \"0\" 
    }," > myfile.json;
done < test;

but once the command is run myfile.json only contains the last line read from the test file. i.e wis_healthpartners

  {
"name": "wis_healthpartners",  
"datatype": "event", 
"searchableDays": 180, 
"maxDataSizeMB": 0, 
"totalEventCount": "0", 
"totalRawSizeMB": "0" 
},

So i think the echo is over writing the lines as the while loop runs and it is left with only the last line. How do i tweak this so that it contains all the lines together? My desired output is as below.

{
  "name": "unix_idx",
  "datatype": "event",
  "searchableDays": 180,
  "maxDataSizeMB": 0,
  "totalEventCount": "0",
  "totalRawSizeMB": "0"
},
{
  "name": "web_pn_iis",
  "datatype": "event",
  "searchableDays": 180,
  "maxDataSizeMB": 0,
  "totalEventCount": "0",
  "totalRawSizeMB": "0"
},
{
  "name": "wis_healthpartners",
  "datatype": "event",
  "searchableDays": 180,
  "maxDataSizeMB": 0,
  "totalEventCount": "0",
  "totalRawSizeMB": "0"

},

Answer 1

In Bash, the > operator intentionally overwrites any existing data in the file, while the >> operator will append.

If you need to make sure the file is empty before you start, you can use printf "" > myfile.json to clear it out before your loop runs, then use >> to continue writing to the end.

Answer 2

If the entries in the test file were properly quoted JSON strings

$ cat test
"unix_idx"
"web_pn_iis"
"wis_healthpartners"

then you could use jq's --slurpfile to pull them into an array, then iterate over that to create an array of objects from a template JSON file:

{
  "name": "",
  "datatype": "event",
  "searchableDays": 180,
  "maxDataSizeMB": 0,
  "totalEventCount": "0",
  "totalRawSizeMB": "0"
}

ex.

$ jq --slurpfile names test '
    . | [foreach $names[] as $name (.; .name |= $name)]' template.json
[
  {
    "name": "unix_idx",
    "datatype": "event",
    "searchableDays": 180,
    "maxDataSizeMB": 0,
    "totalEventCount": "0",
    "totalRawSizeMB": "0"
  },
  {
    "name": "web_pn_iis",
    "datatype": "event",
    "searchableDays": 180,
    "maxDataSizeMB": 0,
    "totalEventCount": "0",
    "totalRawSizeMB": "0"
  },
  {
    "name": "wis_healthpartners",
    "datatype": "event",
    "searchableDays": 180,
    "maxDataSizeMB": 0,
    "totalEventCount": "0",
    "totalRawSizeMB": "0"
  }
]

If you must work with unquoted elements, then you can do the same with --rawfile although it's a bit more work:

jq --rawfile names test '
    . | [foreach ($names | split("\n")[0:-1] | .[]) as $name (.; .name |= $name)]
' template.json

The slice [0:-1] is necessary because split("\n") interprets the test file's trailing newline as indicative of an additional empty element.