Reshaping json / reparing json inside shell script (remove trailing comma)

Question

I've googled around a lot for this and it seems there is no precursor to this need.

I need to edit an applications preference file programatically : as part of a shell script.

and the prefs are stored in strict json format : this means the app loading that preference file will crash at startup if there is a comma , before a closing curly brace }.

normally this wouldn't be an issue.

I'd just use my seds accordingly : if the line containing my faulty text lines up at the end of a section in my example file, then when replacing this text I will always put it without a comma.

If another line containing another faulty bit I want to replace is not at the end, I always replace it including a comma.

Example :

(I use underscore _ as my sed's delimiter because the stings to replace are full of backslashes sometimes)

sed -i 's_"executableDecorator".*_"executableDecorator": "'$user_path'/faf/run \\"%s\\"",_' $user_path/.faforever/client.prefs

if that line was at the end :

sed -i 's_"executableDecorator".*_"executableDecorator": "'$user_path'/faf/run \\"%s\\""_' $user_path/.faforever/client.prefs

this would work, But!...

I have the app end before I run my script so that both aren't editing the preferences at the same time, but even still, because of this app's asynchronous execution the preferences my script will be receiving will differ every time.

it's completely random.

sometimes a line could be in the middle sometimes at the end. The app itself (Java & some json java lib) knows how to append comma or not depending on the context but as part of my shell script... I feel things are going to get bloated.

(If not and there's a shorthand to ensure I have comma or not depending on if next line is }, then that is a better simpler solution that I would be more interested in)

But as it stands I'm looking for a POSIX utility that fixes json so that I can "sanitize" my json prefs file as soon as I'm done butchering it all within my shell script ...does such a thing exist?

EDIT :

here's the base file (whole file, ) :

{
  "mainWindow": {
    "width": 800,
    "height": 600,
    "maximized": false,
    "lastView": "NEWS",
    "lastChildViews": {},
    "x": 67.0,
    "y": 27.0
  },
  "forgedAlliance": {
    "customMapsDirectory": "/home/t/My Games/Gas Powered Games/Supreme Commander Forged Alliance/Maps",
    "preferencesFile": "/home/t/.wine/drive_c/users/t/Application Data/Gas Powered Games/Supreme Commander Forged Alliance/Game.prefs",
    "officialMapsDirectory": "/home/t/faf/./Maps",
    "modsDirectory": "/home/t/My Games/Gas Powered Games/Supreme Commander Forged Alliance/Mods",
    "port": 6112,
    "autoDownloadMaps": true,
    "executableDecorator": "\"%s\""
  },
  "login": {
    "username": "tatsu",
    "password": "*******",
    "autoLogin": true
  },
  "chat": {
    "zoom": 1.0,
    "learnedAutoComplete": false,
    "previewImageUrls": true,
    "maxMessages": 500,
    "chatColorMode": "CUSTOM",
    "channelTabScrollPaneWidth": 250,
    "userToColor": {},
    "hideFoeMessages": true,
    "timeFormat": "AUTO",
    "chatFormat": "COMPACT",
    "idleThreshold": 10
  },
  "notification": {
    "soundsEnabled": true,
    "transientNotificationsEnabled": true,
    "mentionSoundEnabled": true,
    "infoSoundEnabled": true,
    "warnSoundEnabled": true,
    "errorSoundEnabled": true,
    "friendOnlineToastEnabled": true,
    "friendOfflineToastEnabled": true,
    "ladder1v1ToastEnabled": true,
    "friendOnlineSoundEnabled": true,
    "friendOfflineSoundEnabled": true,
    "friendJoinsGameSoundEnabled": true,
    "friendPlaysGameSoundEnabled": true,
    "friendPlaysGameToastEnabled": true,
    "privateMessageSoundEnabled": true,
    "privateMessageToastEnabled": true,
    "friendJoinsGameToastEnabled": true,
    "notifyOnAtMentionOnlyEnabled": false,
    "afterGameReviewEnabled": true,
    "toastPosition": "BOTTOM_RIGHT",
    "toastScreen": 0,
    "toastDisplayTime": 5000
  },
  "themeName": "default",
  "lastGameType": "faf",
  "localization": {},
  "rememberLastTab": true,
  "showPasswordProtectedGames": true,
  "showModdedGames": true,
  "ignoredNotifications": [],
  "lastGameMinRating": 800,
  "lastGameMaxRating": 1300,
  "ladder1v1": {
    "factions": [
      "aeon",
      "cybran",
      "uef",
      "seraphim"
    ]
  },
  "news": {
    "lastReadNewsUrl": "http://direct.faforever.com/2019/03/king-of-badlands-tournament-march-30th/"
  },
  "developer": {
    "gameRepositoryUrl": "https://github.com/FAForever/fa.git"
  },
  "vaultPrefs": {
    "onlineReplaySortConfig": {
      "sortProperty": "startTime",
      "sortOrder": "DESC"
    },
    "mapSortConfig": {
      "sortProperty": "statistics.plays",
      "sortOrder": "DESC"
    },
    "modVaultConfig": {
      "sortProperty": "latestVersion.createTime",
      "sortOrder": "DESC"
    }
  },
  "gameListSorting": [],
  "gameTileSortingOrder": "PLAYER_DES",
  "unitDataBaseType": "RACKOVER",
  "storedCookies": {},
  "lastGameOnlyFriends": false
}

the only part that matters is "forgedAlliance" :

  "forgedAlliance": {
    "customMapsDirectory": "/home/t/My Games/Gas Powered Games/Supreme Commander Forged Alliance/Maps",
    "preferencesFile": "/home/t/.wine/drive_c/users/t/Application Data/Gas Powered Games/Supreme Commander Forged Alliance/Game.prefs",
    "officialMapsDirectory": "/home/t/faf/./Maps",
    "modsDirectory": "/home/t/My Games/Gas Powered Games/Supreme Commander Forged Alliance/Mods",
    "port": 6112,
    "autoDownloadMaps": true,
    "executableDecorator": "\"%s\""
  },

I run commands to obtain this :

  "forgedAlliance": {
    "path": "/home/t/.steam/steam/steamapps/common/Supreme Commander Forged Alliance",
    "installationPath": "/home/t/.steam/steam/steamapps/common/Supreme Commander Forged Alliance",
    "customMapsDirectory": "/home/t/My Games/Gas Powered Games/Supreme Commander Forged Alliance/Maps",
    "preferencesFile": "/home/t/.steam/steam/steamapps/compatdata/9420/pfx/drive_c/users/steamuser/Local Settings/Application Data/Gas Powered Games/Supreme Commander Forged Alliance/Game.prefs",
    "officialMapsDirectory": "/home/t/faf/./Maps",
    "modsDirectory": "/home/t/My Games/Gas Powered Games/Supreme Commander Forged Alliance/Mods",
    "port": 6112,
    "autoDownloadMaps": true,
    "executableDecorator": "/home/t/faf/run \"%s\""
  },

the commands that work (in a standard case where things don't move around) are :

if ! grep -q '"path"' $user_path/.faforever/client.prefs > /dev/null
then
    sed -i '12i"path": "'$user_path'/.steam/steam/steamapps/common/Supreme Commander Forged Alliance",' $user_path/.faforever/client.prefs
    sed -i '13i"installationPath": "'$user_path'/.steam/steam/steamapps/common/Supreme Commander Forged Alliance",' $user_path/.faforever/client.prefs
fi
! grep -q '"preferencesFile": "'$user_path'/.steam/steam/steamapps/compatdata/9420/pfx/drive_c/users/steamuser/Local Settings/Application Data/Gas Powered Games/Supreme Commander Forged Alliance/Game.prefs",' $user_path/.faforever/client.prefs > /dev/null && sed -i 's_"preferencesFile".*_"preferencesFile": "'$user_path'/.steam/steam/steamapps/compatdata/9420/pfx/drive\_c/users/steamuser/Local Settings/Application Data/Gas Powered Games/Supreme Commander Forged Alliance/Game.prefs",_' $user_path/.faforever/client.prefs
! grep -q '"executableDecorator": "'$user_path'/faf/",' $user_path/.faforever/client.prefs > /dev/null && sed -i 's_"executableDecorator".*_"executableDecorator": "'$user_path'/faf/run \\"%s\\""_' $user_path/.faforever/client.prefs

Answer 1

This jq command will make exactly those changes:

jq --arg user_path "$user_path" '
    .forgedAlliance += {
        installationPath: ($user_path + "/.steam/steam/steamapps/common/Supreme Commander Forged Alliance"),
        path: ($user_path + "/.steam/steam/steamapps/common/Supreme Commander Forged Alliance"),
        preferencesFile: ($user_path + "/.steam/steam/steamapps/compatdata/9420/pfx/drive_c/users/steamuser/Local Settings/Application Data/Gas Powered Games/Supreme Commander Forged Alliance/Game.prefs"),
        executableDecorator: ($user_path + "/faf/run \"%s\"")
    }'

This uses

1. --arg user_path "$user_path" to bring the shell variable into the jq program (you could also use the variable binding operator "'"$user_path"'" as $user_path |, but it would involve ugly quote splicing)
2. Update-assignment .forgedAlliance += to process the whole file, updating just the value of the "forgedAlliance" key by merging it with what's on the right.
3. A fresh object constructed from { to } with just the new key values you wanted computed inside it. If there are existing keys with the same name, they will be replaced.
4. $user_path to access that variable binding we made above.

The whitespace is optional - it's just there to make it easier to read on this site.

jq always outputs as valid JSON, so you don't have any comma cleanup to do. You may find the sponge command from moreutils useful for updating the file itself, because there is no -i equivalent in jq, but you can also just redirect to another file

jq ... > tmpfile
mv tmpfile prefs.json

and step around it manually as well.

There is one (slight?) difference to what your code did: you made no changes for path and installationPath if "path" appeared anywhere in the file. There's no way to replicate that with jq directly, but you could split the command in two (one for path, one for all the time) if there's a necessary semantic element to that. This command will always make the change, but if it's already got the same value for a key that doesn't have any effect.

---

If this is a fixed set of replacements, you could also make a file with just the object from point 3 above in it literally (as true JSON, not dynamically computed), and then use

jq --slurpfile tmp rhs.json '.forgedAlliance += tmp[0]'

with the same effect as the big command above.

Answer 2

for strings substitute, put your strings in placeholder A, B below,
escape any / in A with \/, and ~ as sed sub separator rather _
E.g. A -> "executableDecorator":
(B your new inserted string)

sed -E '/A/{N;/A.*\n\s*\}/ {s~(A).*\n~\1B\n~;b} ;s~(A).*\n~\1B,\n~ }' B.json

some expansion instance;

sed -E '/"executableDecorator":/{N;/"executableDecorator":.*\n\s*\}/ {s~("executableDecorator":).*\n~\1B\n~;b} ;s~("executableDecorator":).*\n~\1B,\n~ }' B.json