8
votes
2answers
274 views

Reindexing a large CSV file

I went through the answers in this helpful thread, but my problem seems to be different enough that I can't think of good answer (at least with sed). I have a large CSV file (200+ GB) with rows that ...
3
votes
4answers
102 views

How to remove text matching specific patterns from a file

I want to compare two files to check for differences between them, fileA and fileB. fileA is like a template file and fileB is the file which I want to compare to it. Anytime I find a difference I ...
4
votes
4answers
678 views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
3
votes
3answers
747 views

Merge 2 files based on all values of the first column of the first file

I need to merge below 2 files: file1: TABLES REF-IO HEAD-IO DIFF-IO test 200 500 -300 exam 2 3 -1 final 2 1 1 mail 4 2 2 TOTAL 208 506 -298 ...
4
votes
7answers
562 views

How to split a filename list in 5GB sets?

I have a pretty big list of files which I like to group in smaller list of 5GB each for further processing. The file list is in the following form: "size" (from du) "tab" "absolute path", like this ...
15
votes
1answer
8k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
3
votes
2answers
210 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, ...
1
vote
2answers
402 views

reading from a file and changing its pattern into an array?

I have a CSV file containging entries like this : ipaddress,VLAN,VLANid 10.192.168.1,vlan-xyz,3 10.192.168.1,vlan-abc,8 10.192.168.1,vlan-mnp,11 10.192.163.24,vlan-llz,3 10.192.163.24,vlan-bbz,5 ...
1
vote
4answers
84 views

How to find what device a file is on (and use that in a script)?

I want to find out what device my file is on so that I can use it in a script. I can get this far: $ df . Filesystem 512-blocks Used Available Capacity Mounted on /dev/disk0s2 498438976 ...
2
votes
1answer
233 views

Using awk, how to get desired output from multiple file sources

I have two files: file1: A,D B,E C,F file2: G,H,I J,K,L Using awk, I want to print: A,D B,E,I C,F,L How can I use awk to achieve this?
2
votes
5answers
500 views

Replacing lines in files with file contents

I have several files which contain some PHP includes and I want to substitute them with the file contents. The file looks like foo <?php include("file1.php"); ?> bar baz <?php ...
6
votes
5answers
1k views

Find images by size: find / file / awk

I've been trying to find png image files a certain height (over 500px). I know that file will return image dimensions. Example: $ file TestImg1a.png TestImg1a.png: PNG image data, 764 x 200, ...
5
votes
1answer
2k views

Fast elimination of duplicate lines across multiple files

I have a huge amount of data in which each (data-)line should be unique. There are a lot of files in one folder in which this is already true. It is about 15GB splitted into roughly 170 files with ...
1
vote
2answers
194 views

Splitting giant files into a specific number of lines

I have a CSV that is 40 000 lines long. What is the simplest tool to read from line x to line y? I am sure there are more professional ways to do it but I would just like a bash script that works in ...
9
votes
3answers
6k views

Delete the first n bytes of files

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file ...
8
votes
6answers
238 views

Grab certain contents of a file

So I know tools exist for this problem because I've heard about them, but I don't know what they are. I want to do something like filter out all data but the usernames in /etc/passwd. For example, I ...