0

I am working on post-processing of multi-column log filles, in the following format:

/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_07_lig_cne_420,6, -5.3300, 201.2781, 0,,  26,  8, 1, -0.2132
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_10_lig_cne_420,5, -5.2300, 230.0910, 0,,  26,  8, 1, -0.2092
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_12_lig_cne_420,4, -5.1500, 222.2095, 0,,  26,  8, 1, -0.2060
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_07_lig_cne_420,5, -5.0500, 201.1757, 0,,  26,  8, 1, -0.2020
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_12_lig_cne_420,2, -5.0200, 233.0833, 0,,  26,  8, 1, -0.2008
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_05_lig_cne_420,5, -4.9500, 203.5671, 0,,  26,  8, 1, -0.1980
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_08_lig_cne_420,4, -4.9500, 227.0462, 0,,  26,  8, 1, -0.1980
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_12_lig_cne_420,14, -4.7700, 231.9237, 0,,  26,  8, 1, -0.1908
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_03_lig_cne_420,5, -4.7200, 194.9009, 0,,  26,  8, 1, -0.1888
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_11_lig_cne_420,3, -4.6700, 217.3995, 0,,  26,  8, 1, -0.1868
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_07_lig_cne_420,1, -4.6400, 200.7227, 0,,  26,  8, 1, -0.1856
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_09_lig_cne_420,1, -4.5900, 184.7898, 0,,  26,  8, 1, -0.1836
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_11_lig_cne_420,3, -4.5500, 215.7487, 0,,  26,  8, 1, -0.1820
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_08_lig_cne_420,3, -4.4500, 198.2857, 0,,  26,  8, 1, -0.1780
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_07_lig_cne_420,1, -4.4200, 204.6418, 0,,  26,  8, 1, -0.1768
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_08_lig_cne_420,6, -4.3700, 199.5359, 0,,  26,  8, 1, -0.1748
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_06_lig_cne_420,6, -4.3500, 232.3248, 0,,  26,  8, 1, -0.1740
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_06_lig_cne_420,3, -4.2700, 234.3468, 0,,  26,  8, 1, -0.1708
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_05_lig_cne_420,1, -4.2500, 195.9439, 0,,  26,  8, 1, -0.1700
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_03_lig_cne_420,7, -4.2400, 198.9363, 0,,  26,  8, 1, -0.1696
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_03_lig_cne_420,1, -4.1600, 208.6377, 0,,  26,  8, 1, -0.1664
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_01_lig_cne_420,3, -4.1500, 179.4341, 0,,  26,  8, 1, -0.1660
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_12_lig_cne_420,4, -4.1300, 233.9607, 0,,  26,  8, 1, -0.1652
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_09_lig_cne_420,1, -4.1200, 189.5660, 0,,  26,  8, 1, -0.1648
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_10_lig_cne_420,1, -4.1100, 209.8679, 0,,  26,  8, 1, -0.1644
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_11_lig_cne_420,5, -4.1000, 213.5573, 0,,  26,  8, 1, -0.1640
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_08_lig_cne_420,1, -4.0700, 227.6124, 0,,  26,  8, 1, -0.1628
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_05_lig_cne_420,3, -4.0400, 209.6345, 0,,  26,  8, 1, -0.1616
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_06_lig_cne_420,4, -3.9700, 233.5914, 0,,  26,  8, 1, -0.1588
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_11_lig_cne_420,4, -3.9500, 223.9189, 0,,  26,  8, 1, -0.1580
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_05_lig_cne_420,1, -3.9000, 180.8133, 0,,  26,  8, 1, -0.1560
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_11_lig_cne_420,1, -3.9000, 224.1828, 0,,  26,  8, 1, -0.1560
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_02_lig_cne_420,1, -3.8800, 204.1735, 0,,  26,  8, 1, -0.1552
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_09_lig_cne_420,1, -3.8500, 195.5399, 0,,  26,  8, 1, -0.1540
/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/7000_cne_lig420.AllBoxes/7000_10_lig_cne_420,2, -3.8400, 227.9037, 0,,  26,  8, 1, -0.1536

Note that the columns 1 and 2 separated by comma (,), while the rest column by comma space(, ). From this log file I need to:

  1. replace all data in the first column (long unix-format pathway /Users/gleb/Desktop/scripts/...) with the corresponding line number (just line N);
  2. remove enterily columns 6-9 ( the four last columns);

Eventually the resulting log should contain the same number of rows but only from the column 1 (with substitution!) to 5 (the last column with 0,).

What I have been able to accomplish is the substitution within the first column using sed, which however just cut the pathway but does not introduce corresponded line number there:

sed -i '' -e 's|\/Users/gleb/Desktop/scripts/analys_clusters/sub_folders_to_analyse/*.*/||' log.txt
0

1 Answer 1

1
gawk -F'^[^,]*,|, ' '{ print NR, $2, $3, $4, $5; }' OFS=', ' infile

to skip first N line(s), adds NR> N to the awk, so first N lines will be skipped; to skip first line, you would do:

gawk -F'^[^,]*,|, ' 'NR> 1{ print NR, $2, $3, $4, $5; }' OFS=', ' infile

Subsequently you will need to modify NR to NR-1, so it will start from 1 not 2, or just replace it with another temp variable like:

gawk -F'^[^,]*,|, ' 'NR> 1{ print ++lineNumber, $2, $3, $4, $5; }' OFS=', ' infile

^[^,]*, matches from the beginning of the line up-to first comma character;
, matches a comma-space characters.

these above we defined as filed separators (separated with |), and based on that we printed corresponding fields; NR in awk represents the current line number.


Another option is using cut and nl:

<infile cut -d',' -f2-6 |nl -w1 -s', '

cut command cuts the fields 2~6 and nl numbers the lines with comman-space separated , ; -w sets 1 width column to numbers.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .