I have a large (~900MB) tab-delimited text file that I will be processing in a downstream program. I need to delete any line with a missing value. The correct number of columns are on each line (so a missing value would correspond to 2 tabs).
Note: My actual data has ~2 million lines and 80-300 columns. Possible characters are a-z A-Z 0-9 - (hyphen) _ (underscore) and tab (delimited). No spaces or special characters are in the file.
I am new to this kind of scripting, so an explanation of any code provided would be appreciated. I normally use R, but my filesizes have outgrown the data manipulation functionality of R.
How can I at the terminal (or within a shell script) delete lines with missing values from a file (e.g. using
Example Input File:
Col1 Col2 Col3 A B C D F G H I J K
Example Output File:
Col1 Col2 Col3 A B C G H I