Tag Info

New answers tagged

0

You should also consider using find - another very useful approach - e.g. look at: find . -name "*csv" -exec grep "" {} + | sed -e 's/:/,/g' -e 's/\.\///g' This is very similar to the questions described here: Concatenate filename inside of CSV file for each line for multiple CSV files? See also: ...


0

This is an off the cuff answer: for dir in $(ls ); do echo " $dir - this is the directory"; for csv in $(ls $dir/*csv); do cat $csv|sed -e "s/\(.*\)/$dir, \1/g"; done; done You could then add options to specify file names, e.g. for myfile in EA_sing EA_ska EA-tat; do for csv in $(ls $dir/$myfile*)... and then pipe them to a file of your choice, e.g. ...


5

This should do the trick: awk -F '[<>]' ' NR!=1 && FNR==1{printf "\n"} FNR==1{sub(".*/", "", FILENAME); sub(".xml$", "", FILENAME); printf FILENAME} /double/{printf " %s", $3} END{printf "\n"} ' $path_to_xml/*.xml > final_table.csv Explanation: awk: use the program awk, I tested it with GNU awk 4.0.1 -F ...


0

It is a bit unclear what you are asking, I assume you want to print for every row max value of the row and column header for column in wich this value is found: BEGIN { FS = ","; } NR == 1 { for (i = 1; i <= NF; i++) { x[i] = $i; } next; } { max = $1 + 0; for (i = 1; i <= NF; i++) { if (max <= ($i + 0)) { ...


0

The following allows for a repeating maximum on the same line. awk -F, 'NR==1 { split($0,head,FS); next } { max=0; delete a; for(i=1;i<=NF;i++) if($i>=max){ max=$i; a[max]=a[max]head[i]" ("i"), " } print "max " max "\t" substr(a[max], 0, length(a[max])-2) }' file input: hdr A,hdr B,hdr C,hdr D,hdr E,hdr ...


0

The problem with CSV is it doesn't parse nicely with normal shell tools. They simply don't do it nicely. It can be done in trivial cases, but really - a scripting language is the tool for the job. I'd be thinking more perl personally: #!/usr/bin/env perl use strict; use warnings; use Text::CSV; my $csv = Text::CSV->new(); open ( my $input, "<", ...


0

Your future self (and anyone else having to maintain the software) will thank you if you use a language like Python for this. Of course it's not going to be a one-liner, but at least it's readable Naive pseudo-code goes something like this (completely untested): import csv import defaultdict with open('max1.csv') as file_handle: csv_reader = ...


4

Regarding glob expansion possibly exceeding a limit - yes and no. The shell's already running, and so it won't stop. But if you were to pass the entire globbed array as arguments to a single command, then yes, that is a definite possibility. The portable and robust way to handle this involves find... find . \! -name . -prune -name pattern -type f -exec cat ...


0

You can try this single liner for each file. The awk multiple delimiters does efficient splitting and tr concats all lines in memory , rather than on disk. for f in `ls *.xml` ; do echo $f,`grep double $f | awk -F '[<>]' '{print $3}' | tr '\n' ','`; done I can not profile this at my end - since I do not have the same data, but my hunch is ...


-1

You’re writing twice for every file. This is probably the most expensive part. You’ll instead want to try to keep the whole thing in memory, probably in an array. Then write once in the end. Look into ulimit if you start hitting memory limits. If you’re increasing this workload to 10-100x, you’re looking at maybe 10-100 GB of memory. You could batch this in ...


1

Two steps: create a remover script (AUX) with print unless m!gmail.com!hotmail.com|...! (the regular expressio is huge but there is no problem) apply it to All.txt Code: perl -n0E 's/\n/|/g; say "print unless m!\\b($_ç)\\b!\n" ' remove.txt > AUX perl -n AUX All.txt > outfile Update1: to be case-insensitive add a iin the match operator: ...


1

{ sed -ne's/./^[^,]*,[^,]*&/p' | grep -vf- ./All.txt } <./remove.txt >./outfile Is what I think you are asking about. I'm not sure how it is relevant to ruby or to the line of data you're talking about... If you want the matches to be case-insensitive then just add the -ignore case option to grep like: { sed ...


1

Another sed: sed '$!N;/\n,/s/\([^,]*\).*\n/&\1/;P;D' <in >out For each input line which is ! not the $ last, sed will append the Next input line to pattern space as preceded by a \newline character. It will then attempt a s///ubstitution which involves copying the first possible group of ^, not-comma characters to the space just preceding a ...


1

Perl solution: perl -F, -le '$, = "\t"; print @F[1,5,6,7] if $F[5] > 4 || $. == 1' file -F, specifies the pattern to split on. -F implicitly sets -a -a turns on autosplit mode when used with a -n. An implicit split command to the @F array is done as the first thing inside the implicit while loop produced by the -n. -a implicitly sets -n -n ...


1

With sed: sed '/^[0-9]/{ # if line starts with digit h # overwrite hold buffer with pattern space content s/\([^,]*\),.*/\1/ # extract timestamp x # exchange: put the original line back into pattern } # space and the timestamp in hold space /^,/{ ...


0

To get around your issue with included commas in the fields, change your field separator to quote+comma; assuming that your included commas in the fields are not at the very beginning and/or end of the field. $ awk -F'(\",)' '{print $1 $2 $17}' test.txt Just make sure you escape the quote and surround field separator in single quotes to protect from your ...


2

There's a little-known program named csvquote that makes it possible to use standard tools like cut, sed, and awk to handle CSV files. It works by mapping special characters inside quotes to some unprintable characters, and then mapping them back. With this program, it's as simple as this: csvquote file.csv | cut -d , -f 1,2,5 | csvquote -u Output: ...


3

cut -d\" -f2,4,10 <in | tr \" , >out ...would grab only the quoted bits for fields 1,2 and 5 and afterward ensure they get separated w/ commas. Or, since it appears some commas can occur within the quotes... cut -d\" -f-5,10-11 <in | sed s/,$// >out ...or even... cut -d\" -f-5,10 <in | paste -d\" - /dev/null >out ...might set that ...


1

awk -F',' '{print $1 $2 $5}' - Is that what you are looking for ?


1

You can use awk and set the field separator to either spaces or comma. Then, loop in blocks of 2, printing the first and second fields together with blocks of two: $ awk -v FS='(\\s+|,)' '{for (i=3; i<=NF; i+=2) print $1, $2, $i, $(i+1)}' file 3923 001 L05 LV 3923 001 L05 RM 3923 002 L12 RA 3923 002 L12 LA 3923 003 I06 ALL 3923 004 G04 RV 3923 004 Z09 ...


1

Simplyfied variant of User112638726 answer sed ':1;s/\(\(.*\s\s\+\)[^,]\+\),/\1\n\2/;t1' sed -r ':1;s/((.+\s\s+)[^,]+),/\1\n\2/;t1' will choice repeated part as «any symbols before 2(or more) spaces». Then you can pipe output through uniq -s 10


6

Given the format of your example this should work for any number of comma separated strings after the initial large space(if it's a tab just change the spaces in the second s/// to \t sed ':;h;s/,.*//;p;x;s/ [^,]*,/ /;t;d' file 3923 001 L05 LV 3923 001 L05 RM 3923 002 L12 RA 3923 002 L12 LA 3923 003 I06 ALL 3923 004 G04 RV 3923 ...


2

Possible solution with awk: awk -F" " '{ x = $3 " " $4 " " $5; split(x, a, ","); for (i in a) { print $1, $2 "\t" a[i]; } }' file Output should be: 3923 001 L05 LV 3923 001 L05 RM 3923 002 L12 RA 3923 002 L12 LA 3923 003 I06 ALL 3923 004 G04 RV 3923 004 Z09 ALL And if your input contains multiple , as in comments: 3923 001 L05 ...


3

It is possible using sed. Pipe the input to the below. | sed 's/\(^.\{12\}\)\([^,]\+\),\([^,]\+\)/\1\2\n\1\3/' output 3923 001 L05 LV 3923 001 L05 RM 3923 002 L12 RA 3923 002 L12 LA 3923 003 I06 ALL 3923 004 G04 RV 3923 004 Z09 ALL Here is the source to handle input with multiple "," For original input without tab, use | perl ...


2

Nested if statements are effectively statement and statement, so if you do not need to do any particular processing as you step through the nesting, then you can just join them all up with &&. awk '{ if( ( $4 == "TRX" || $4 == "TX" ) && ( $10 == "BTS INT UNAFF" || $10 == "LOCAL MODE" ) && ( $12 != ...


2

If I understand you correctly, you want to create a single (100GB-sized?) file from all of your files. In that case cat might do the trick: cat foo1 foo2 bar1 bar2 > newfile or in your case cat folder1/* folder2/* > newfile which will take all files from folder1 and folder2 and merge them (as text) into newfile. note that cat just appends the ...


1

Try something like this: find /path/to/dir1 /path/to/dir2 -type f -name '*.csv' -exec cat {} + >/path/to/merged.csv Make sure not to put the merged file in either of the input directories. :)


0

From The AWK Programming Language: The simplest awk program is a sequence of pattern-action statements: pattern { action } pattern { action } ... So, you can try following: awk -F, '$7 == "Active" { x += $1; }; $6 == "value6" && $5 != value5 { y += $1; }; END { print x, y; }' file


2

The lines in your CSV don't have the same number of fields. Happily ignoring that issue however, you could do something like this: awk -F, -v OFS=, '{ if($1 ~ /^[ \t]*$/) $1 = old; else old = $1 } 1' file.csv | \ datamash -H -t, -g 1 sum 4 datamash above is GNU datamash. awk normalises the first column, and datamash does the grouping and summing. ...


1

Something like: awk -F, '{ # print first three columns printf("%s,%s,%s", $2,$1,$3); #for all other columns for ( i = 4; i < NF; i++ ) { # if column number every third if ( ( i - 4 ) % 3 == 0) { printf(",%s", $i); } } #print newline print ""; }' your_file.csv



Top 50 recent answers are included