0

When using grep (and sed also) the results I get in the terminal are different than what I get if I direct the output to file. My starting file is the captured output from a darknet yolo mAP run and looks like this:

     CUDNN_HALF=1 
net.optimized_memory = 0 
mini_batch = 1, batch = 32, time_steps = 1, train = 0 
Create CUDA-stream - 0 
 Create cudnn-handle 0 
nms_kind: greedynms (1), beta = 0.600000 
nms_kind: greedynms (1), beta = 0.600000 
nms_kind: greedynms (1), beta = 0.600000 

 seen 64, trained: 96 K-images (1 Kilo-batches_64) 

 calculation mAP (mean average precision)...
 Detection layer: 139 - type = 28 
 Detection layer: 150 - type = 28 
 Detection layer: 161 - type = 28 

 detections_count = 3688, unique_truth_count = 2114  
 rank = 0 of ranks = 3688 
 rank = 100 of ranks = 3688 
 rank = 200 of ranks = 3688 
.
.
.
 rank = 3400 of ranks = 3688 
 rank = 3500 of ranks = 3688 
 rank = 3600 of ranks = 3688 
class_id = 0, name = car, ap = 87.89%        (TP = 1414, FP = 130) 
class_id = 1, name = motorcycle, ap = 90.91%     (TP = 10, FP = 0) 
class_id = 2, name = truck, ap = 96.37%      (TP = 270, FP = 33) 
class_id = 3, name = gcp, ap = 96.08%        (TP = 305, FP = 13) 

 for conf_thresh = 0.25, precision = 0.92, recall = 0.95, F1-score = 0.93 
 for conf_thresh = 0.25, TP = 1999, FP = 176, FN = 115, average IoU = 83.92 % 

 IoU threshold = 75 %, used Area-Under-Curve for each unique Recall 
 mean average precision ([email protected]) = 0.902287, or 90.23 % 

Set -points flag:
 `-points 101` for MS COCO 
 `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) 
 `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset

I have tried numerous arrangements and still get the same results when I try to pull out the class_id result lines e.g.

grep class_id results.txt > class_results.txt

which results in the following text in the file:

 rank = 0 of ranks = 3688 
 rank = 100 of ranks = 3688 
 rank = 200 of ranks = 3688 
.
.
.
 rank = 3400 of ranks = 3688 
 rank = 3500 of ranks = 3688 
 rank = 3600 of ranks = 3688 
class_id = 0, name = car, ap = 87.89%        (TP = 1414, FP = 130) 
class_id = 1, name = motorcycle, ap = 90.91%     (TP = 10, FP = 0) 
class_id = 2, name = truck, ap = 96.37%      (TP = 270, FP = 33) 
class_id = 3, name = gcp, ap = 96.08%        (TP = 305, FP = 13) 

If I leave off the > class_results.txt part the output looks correct (or as I expect and desire):

class_id = 0, name = car, ap = 87.89%        (TP = 1414, FP = 130) 
class_id = 1, name = motorcycle, ap = 90.91%     (TP = 10, FP = 0) 
class_id = 2, name = truck, ap = 96.37%      (TP = 270, FP = 33) 
class_id = 3, name = gcp, ap = 96.08%        (TP = 305, FP = 13) 

I have tried using sed to remove the lines containing 'rank' i.e. sed '/rank/d' class_results.txt > class_results_concise.txt but it not only removes the line containing 'rank', it also removes the class_id = 0 line which I want to keep in the output. I have tried the in place editing form for sed and get the same thing.

How do I just output the lines containing class_id to a text file?

This is all on Ubuntu 20.04. I'm guessing I'm misunderstanding something about stdout etc. Will be much grateful for a solution. Many thanks in advance.


Additional info: Upon examining the ASCII I see that the lines containing the word 'rank' are preceded by a CR (ASCII 13) whereas the lines with class_id are preceded by an LF (ASCII 10). I suspect I need a way to convert the CRs to LFs???

5
  • 2
    Looks to me like the rank = X of ranks = Y line is an updating status report, with each new line of output intended to replace the previous. Can you get your command to run "silently" (less verbosely) so that it doesn't produce these status report lines? Feb 26, 2022 at 13:52
  • There doesn't appear to be anyway to control the text output other than to switch it on or off. I don't understand why this would matter since I am capturing the output to text file and post processing it with grep or whatever. I suppose I could examine the ASCII to see if there are special characters being used that confuse grep etc.
    – tinyMind
    Feb 26, 2022 at 14:13
  • To convert all CR to NL, pipe the stream through tr '\r' '\n'. Feb 26, 2022 at 15:18
  • great thanks, if you want to make that a proper SE answer I'll test it and like yours if it works.
    – tinyMind
    Feb 26, 2022 at 15:36
  • I mean choose it (and it solves my problem BTW)
    – tinyMind
    Feb 26, 2022 at 15:53

1 Answer 1

0

Thank you to @Paul_Pedant for the tr tip.

piping the file through tr '\r' '\n' before running grep solves the problem. It seems that the output text file I was trying to filter had CR in some places and LF in others. These seem to be treated differently by programs like grep. Ed Morton suggests that sed 's/\r$//' would be a better solution in general however, as per his comment:

tr '\r' '\n' which would convert ALL \rs in your input to \ns but it's perfectly valid for \rs to appear elsewhere in input and if they did then that tr would tear input lines apart. The right solution to unwanted \rs at the end of lines is to simply remove \rs at the end of lines using, e.g. sed 's/\r$//' instead.

13
  • That's the opposite of the advice @Paul_Pedant gave you but actually I wouldn't use tr for this anyway as it does too much and in doing so can corrupt your data. Please read stackoverflow.com/q/45772525/1745001 for more info on this problem and recommendations on how to solve it simply and robustly.
    – Ed Morton
    Feb 26, 2022 at 22:05
  • I'm more than slightly perplexed by this response. What he suggested really did solve my problem. Passing the output text file from darknet through tr then allowed it to be processed perfectly with grep. From what I can tell Windows file format has no bearing on this issue other than in coincidence. As far as I understand darknet is not a Windows based framework, but a Linux/Unix one.
    – tinyMind
    Feb 27, 2022 at 2:03
  • Windows-generated or not, your problem as described in the comments appears to be \rs at the end of lines. Paul suggested tr '\r' '\n' which would convert ALL \rs in your input to \ns but it's perfectly valid for \rs to appear elsewhere in input and if they did then that tr would tear input lines apart. The right solution to unwanted \rs at the end of lines is to simply remove \rs at the end of lines using, e.g. sed 's/\r$//' instead. If using tr seemed to solve your problem then maybe you don't have \rs elsewhere in your input or you just haven't noticed breakage yet...
    – Ed Morton
    Feb 27, 2022 at 9:07
  • ...but if you just use sed instead of tr then whether or not \rs appear elsewhere is simply not a concern. Also, your answer says you used tr '\n' \r' instead of tr '\r' \n' which is why I mentioned that being the opposite of Pauls suggestion (and idk why that'd work).
    – Ed Morton
    Feb 27, 2022 at 9:08
  • ah I made an error in typing the the tr command params. Corrected that now cheers. I know '\r' is used when you want to overwrite a line of text in stdout, which is what I guess was going on here. Not sure what other uses there are of this special char??
    – tinyMind
    Feb 27, 2022 at 15:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .