New answers tagged

0

As first step run through your CSV file on and convert to Unix line endings: dos2unix your_csvfile This changes \r\n to \n Then in the next step : Using GNU sed we keep track of even/odd numbers of double quotes as shown here: sed -Ee ' h;s/[^"]*//g /^(..)*$/!{ z;G;N;D } g;s/\n//g ' your_csvfile perl -pe 's/\n/<>/e while y/"// % ...


0

Using Raku (formerly known as Perl_6) perl6 -MText::CSV -e 'my $csv=Text::CSV.new; .perl.put for $csv.getline_all(open($*ARGFILES, :r, :!chomp));' Sample Input: ID, Name, text "1","abc","Line 1" "2","def","Line2 ""line2"",line2" "3","ghi","line3" ...


0

Using Raku (formerly known as Perl_6) raku -MText::CSV -e 'my $csv=Text::CSV.new; .perl.put for $csv.getline_all(open($*ARGFILES, :r, :!chomp));' Sample Input: ID,Code,Message,date 1244,,"""Exception error : java connection error :8080 Connection refused""",01-09-2021 Sample Output: $["ID", "Code", "...


2

If your CSV was generated by MS tools such as Excel then the "newline" in the middle of the field is a LF alone while the "newline" at the end of each record is a CRLF like this (note the LF $ mid-field vs the CRLF ^M$ at the end of the records): $ cat -Ev file ID,Code,Message,date^M$ 1244,,"""Exception error : java ...


2

If you have csvkit utilities you can fix up lines with embedded newlines, for example by transforming the newline into the literal two characters \n: csvformat -M $'\r' datafile | # temporarily end lines with $'\r' sed -e ':a' -e 'N;$!ba' -e 's/\n/\\n/g' | # transform $'\n' into '\n' tr '\r' '\n' #...


1

One simple way is to just remove newline characters on lines with only 3 fields: $ perl -F',' -pane 's/\n// if $#F==2' file ID,Code,Message,date 1244,,"""Exception error : java connection error:8080 Connection refused""",01-09-2021 1245,,"""Exception error :""",01-09-2021 1246,,"ffadsdasd&...


1

I have found a way to quickly compare 2 files with 1 Million rows. My need was that the two files have to be equal. The diff command was slow but to make it faster just sort the files before comparing them. So basically: sort file01.txt > file01_sorted.txt sort file02.txt > file02_sorted.txt Then run the diff command: diff file01_sorted.txt ...


1

I just noticed that each column of your data actually begins with the name of the column. I must have missed that when I first saw your question. That makes it not only possible but fairly easy to reformat the data. #!/usr/bin/perl use strict; my @headers; # array to hold the headers in the order they were seen. my @search; # array to hold a copy of @...


0

Or use csvformat from the csvkit -- this tool takes care of quoting any field containing the delimiter: I added a line to the input file that has tabs in it. $ cat a.txt a,"test, part2 ""the start""",b c,d,e with tabs $ csvformat -D $'\t' a.txt a "test, part2 ""the start""" b c d "e ...


3

With mlr: mlr -N --icsv --otsvlite cat < file.csv > file.tsv Or: mlr -N --c2t --quote-none cat < file.csv > file.tsv But note that if a csv field contains a tab character, it will end up not escaped in the output and therefore introducing an extra field. With GNU sed, you could do the same with: sed -E ' # append next line as long as there is ...


3

bash 5.1 comes with a loadable CSV module BASH_LOADABLES_PATH=${BASH/\/bin\//\/lib\/} enable -f csv csv csv -a fields "$line" new_line=$(IFS=$'\t'; echo "${fields[*]}") declare -p line fields new_line outputs declare -- line="a,\"test, part2 \"\"the start\"\"\",b" declare -a fields=([0]="a&...


1

Your rewritten question is now conceptually pretty easy to answer: You have an array of tags that may or may not be present in each row of data. You want to read in each row, and go through the columns in order, checking to see if the tag in that column is the one that's expected. If not, insert a blank cell, and check the next column. Once you get to ...


0

Well, assuming all the fields are separated with a single space you can try this: while read SAMPLE REGION REST; do OUT=$(sed s/'[",;]'//g <<<$REGION);echo "$SAMPLE $REGION $REST" >>$OUT.csv; done < all.csv If there is no space between the fields, you can change the internal field separator by using IFS= as folowing example: ...


0

Something like this should work. You can use awk to seperate and create the files and then p7zip for encryption after. for f in file; do awk ' /Alabama/ { print > "Alamaba.txt" } /Oklahoma/ { print > "Oklahoma.txt" }' $f 7z a -p -mx=9 -mhe -t7z states.7z Alabama.txt Oklahoma.txt #7z a -p -mx=9 -mhe -t7z Alabama.7z ...


Top 50 recent answers are included