2

Stadard unix command line tools (grep, cut, sed etc.) all work one line at a time. And nearly always, this is great.

However I am trying to parse some postgresql slow query logs. Each entry has some stuff at the start (datetime, duration), and then the SQL query. The SQL query might have newlines, so each "entry" in the log file can take more than 1 line (since the newlines in the query are put into the log file directly, and not escaped). I'd like to somehow "merge" these lines together, so that 1 line = 1 log file entry. Now, sometimes one entry is wholly on one line, and sometimes one entry is spread across up to 10 lines.

Is there a unix tool that can somehow "lineify" this file? I'd like to give it a (PCRE) regex and it will split the line/stdin based on that. All actual newlines between occurrences of this regex should be replaced with "\n" or something I can specify.

There is probably one liners to do this with perl, but I'd like to see if anyone has already made this programme before I make it myself.

UPDATE: I could provide sample data, but I'd like to know the generic problem. SQL Servers can potentially make multi-line log files. I want a generic solution to turn any file into a newline separated unix-y style file.

  • 2
    It would be much easy to help you if you supply us with some example data(especially for single and multy lines query). – Costas Apr 7 '15 at 14:47
1

Using gawk you could use a (subset of) PCRE expression as register separator (RS), define a different output register separator (ORS) and replace \n.

Example:

gawk 'BEGIN {RS="[ ]*;\n"; ORS="\n===\n"}
            {gsub("\n","\\n");   print} '

in this example:

  • registers are separated by [ ]*;\n in the input
  • registers are separated by "\n===\n" in the oputput
0

Parse the logfile line for line and suppress all \n's. When you see a new entry, first write \n except for the first time.
You said Each entry has some stuff at the start (datetime, duration) but you did not give an example. Ok, I will call it NEW_ENTRY, you can modify.

inStatement=0
cat logfile | while read -r line; do
   if [[ ${inStatement} = 0 ]]; then
      inStatement=1
   else
      [[ ${line} = NEW_ENTRY* ]] && echo
   fi
   echo -n "${line} "
done
echo

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.