Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [large-files]

The tag has no usage guidance.

1
vote
1answer
24 views

Locate Large Files - Discrepancy between Parent Folder and Child Folder

System: Red Hat Enterprise Linux Server release 7.6 (Maipo), 3.10.0-957.12.1.el7.x86_64 GOAL: Find files that are large in size to see how they can be remediated. Note: Folders with "application" in ...
1
vote
0answers
31 views

I have a directory of 110 large files. I want to copy them in the batch of n (e.g., 10) to aws s3 [closed]

I have 110 large files inside a directory. But I cannot copy them at once since the system gets timed out after 50 min. I want to copy files 1 to 10, next 11 to 20, next 21 to 30, and so on. The order ...
4
votes
4answers
284 views

Fast way to extract lines from a large file with 80 billion lines

I have a large file with 80 billion lines. Now I want to extract a few lines (around 10000) which I know the line number, what is the fastest way to deal with it. Is it possible to extract those ...
3
votes
1answer
195 views

What filesystem allows transferring files between Linux and OS X?

I have a file that is 12 GB that I am trying to copy from a MacBook Air to a Debian computer, using a USB. I tried formatting the USB in many different ways, such as NTFS, FAT32, OS X Journalizing but ...
3
votes
1answer
609 views

How copytruncate actually works? [closed]

we would like to understand copytruncate before rotating the file using logrotate with below configuration: /app/syslog-ng/custom/output/all_devices.log { size 200M copytruncate dateext ...
0
votes
1answer
115 views

Comparing two files with an easy readable GUI program?

I would like to compare two large files that are not close to 1Gb in size. Kompare crashes when the files load is large. I configured Kompare as in the photo below: I would like to present the ...
7
votes
4answers
8k views

How to find duplicate lines in many large files?

I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free. My goal: I want to find all all duplicate lines across ...
126
votes
14answers
24k views

Replace string in a huge (70GB), one line, text file

I have a huge (70GB), one line, text file and I want to replace a string (token) in it. I want to replace the token <unk>, with another dummy token (glove issue). I tried sed: sed 's/<unk&...
2
votes
1answer
2k views

Why does head; tail on a large file sometimes take a long time and sometimes not?

I am writing a bash script that reads the first 10 lines and the last 10 lines of a .txt file. It looks for started (head) and completed (tail) and compares the number of occurrences using grep. The ...
4
votes
4answers
353 views

grep: Which *patterns* are matched, not which text?

In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that. In other words, given a fairly large ...
6
votes
1answer
502 views

best way to split huge files based on a field when awk is too slow

I am having problems dealing with huge .gz files (greater than 500G). My goal is to split each of these by the 4th field within these files. There is this beautiful awk one-liner I have used before to ...
0
votes
2answers
437 views

Awk - Compare Value from a column with a variable, count how many times it passes for each ID

I have been trying to find a solution but now I need some help. First, as input, I have a Big File (5.5G) with this structure: scaffold4691_size302 2 T 1 scaffold4691_size302 3 ...
4
votes
2answers
5k views

diff two large CSV files (each 90GB) and output to another csv

This is a follow up question from Sort large CSV files (90GB), Disk quota exceeded. So now I have two CSV files sorted, as file1.csv and file2.csv each CSV file has 4 columns, e.g. file 1: ID Date ...
7
votes
1answer
965 views

Sort large CSV files (90GB), Disk quota exceeded

Here is what I do right now, sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv the file is 90GB,I got this error message sort: close failed: /some_dir/sortmdWWn4: Disk quota ...
1
vote
2answers
1k views

fast ways of removing beginning lines from large text file

I have a big text file (>500GB), all the ways I can find (sed/tail and others) all require write the 500GB content to disk. Is there anyway to quickly remove the first a few lines in place without ...
1
vote
1answer
357 views

Is there an efficient way to parse blocks of text in python?

I have an enormous file (~70GB) with lines that look like this: $ cat mybigfile.txt 5 7 1 1 0 -2 0 0 2 0 4 0 -4 0 0 4 0 0 1 -1 0 0 0 ...
0
votes
1answer
50 views

Using split files without merging them

I'm thinking a way of storing large files on vfat file system. Obviously, the only way to store files larger than 4GiB is to split them. I'm aware that we can use split and cat commands for splitting ...
0
votes
1answer
542 views

Opening big storage log file in vi

I have a Storage log in an AIX partition. When I try to edit the file with a vi command, I receive the following message: ex: 0602-101 Out of memory saving lines for undo delete lines I don't want to ...
4
votes
1answer
521 views

Recommended maximum percentage to fill a large ext4 data drive

The default reserved blocks percentage for ext filesystems is 5%. On a 4TB data drive this is 200GB which seems excessive to me. Obviously this can be adjusted with tune2fs: tune2fs -m <...
1
vote
1answer
67 views

Word frequency gawk memory leak

I have the following bash script: #!/usr/bin/env bash grep -e '^[a-zA-Z]\{4,8\}$' data/words3.txt | tr '[:upper:]' '[:lower:]' | sort -u > data/passphrase-words.txt function wordfrequency() { ...
1
vote
1answer
252 views

Does traditional vi (not vim) have large file support?

Is there a way to compile traditional Vi (ex) with large-file (3+ GB) support? I'm trying to open a file that is around 3.5GBs in size and Vi simply says: Tmp file too large. If large-file support ...
3
votes
2answers
766 views

How to determine size of tar archive without creating it?

I'm archiving a few directories every night to LTO-7 tape with about 100 or so large (2GB) files in each of them. As a check that the data has been written correctly, I'm verifying that the number of ...
2
votes
1answer
349 views

Faster (uncompressed) archiving tool than tar?

I'm using tar to archive a bunch of files to LTO-7 tape. Generally each file is about 1-2GB, and there can be several hundred of them in each archive (up to ~1TB per archive). At the moment, I'm ...
1
vote
2answers
118 views

Print each line multiple but different times

I have a huge file, ~4000000 lines that look like that: file: 1 1 2 3 2 4 ... basically, each line contains a number. Considering this example I would like to generate a file that will look ...
7
votes
7answers
3k views

grep with pattern from one file (3.2Gb) matching in another file (4.8Gb)

I have two text files. One is a text file with name, email address and other fields. Some lines from file1: John:myemail@gmail.com:johnson123:22hey Erik:thatwhatsup@gmail.com:johnson133:22hey Robert:...
3
votes
1answer
2k views

Transfer Size limit for SCP, SFTP, RSYNC?

Is there a maximum size limit to the size of collective size of the files that are to be transferred in one go for the above protocol. My intent is to know if for example after 1 GB any of these ...
0
votes
3answers
230 views

How to loop through a file and make each line a new regular expression into an awk statement?

FileA.txt: ATGCATGC GGGGGGTT TTTTT AAAA FileB.txt: asdfasdf blah2 ATGCATGC blah3 blah4 delte-me-too GGGGGGTT blah5 blah5 .... I want to compare the each line from FileA.txt and check if it is ...
19
votes
3answers
39k views

Replace text quickly in very large file

I have 25GB text file that needs a string replaced on only a few lines. I can use sed successfully but it takes a really long time to run. sed -i 's|old text|new text|g' gigantic_file.sql Is there a ...
0
votes
1answer
639 views

“Cannot reallocate” when create file?

I am trying to create excel sheet based on multiple files on a root. I read files line by line and append in the final excel sheet. I am trying this shell script on small files and it worked 100%, ...
0
votes
1answer
2k views

aix: large files

/opt/freeware/bin/ls //media/nfs/slack64/pubblica /opt/freeware/bin/ls: cannot open directory //media/nfs/slack64/pubblica: Value too large to be stored in data type This happen with coreutils-8.23 ...
-2
votes
4answers
1k views

Extract several lines from large text file [duplicate]

I have a large text file. I need to quickly pull a bunch of lines, say from #14600 to #14700, from this file, as a separate file. How it could be done?
2
votes
4answers
6k views

Fast way to add/append to a large file [closed]

I have a bash script that reads a rather large file, line by line, and for each line does some processing and writes the results to another file. Currently I am using echo to append to the end of the ...
19
votes
2answers
2k views

Why are these files in an ext4 volume fragmented?

I have a 900GB ext4 partition on a (magnetic) hard drive that has no defects and no bad sectors. The partition is completely empty except for an empty lost+found directory. The partition was formatted ...
3
votes
3answers
3k views

how to create multi tar archives for a huge folder

I have a large folder with 30M small files. I hope to backup the folder into 30 archives, each tar.gz file will have 1M files. The reason to split into multi archives is that to untar one single large ...
3
votes
3answers
1k views

deduplication of lines in a large file

The size of the file is 962,120,335 bytes. HP-UX ******B.11.31 U ia64 ****** unlimited-user license hostname> what /usr/bin/awk /usr/bin/awk: main.c $Date: 2009/02/17 15:25:17 $Revision: ...
26
votes
3answers
41k views

Transferring large (8 GB) files over ssh

I tried it with SCP, but it says "Negative file size". >scp matlab.iso xxx@xxx:/matlab.iso matlab.iso: Negative file size Also tried using SFTP, worked fine until 2 GB of the file had transferred,...
2
votes
2answers
71 views

Create small record files from a big group of record file

I have a file like this : Mainfile: Record|1111|ABC text in between for record 1 text in between for record 1 Record|2222|XYZ text in between for record 2 Record|3333|XYZ ...
1
vote
2answers
1k views

Read file and find all occurrences and generate hash for the content between the occurrences

I have a file like this : 1 Record|1111|ABC 2 text in between for record 1 3 text in between for record 1 4 Record|2222|XYZ 5 text in between for record 2 6 Record|3333|XYZ 7 text ...
2
votes
1answer
1k views

How to handle repetitive download of some specific large files that come from the internet

We are going to offer an App which will allow our customers to download some sort of audio guide for the castle I work for. I would really like to not have every user who downloads the App (through ...
1
vote
1answer
688 views

Server reports incorrect file size when downloading a large file

I have a large tarball file on an OS X web server that I uploaded via sshfs. The file on my computer prior to uploading is 5345834658 bytes. The file after copying is 5345834658 bytes on the server so ...
7
votes
3answers
2k views

How to recompress 2 million gzip files without storing them twice?

I have about 2 million (60GiB) of gzipped small files and I would like to create a compressed archive containing all of them in an uncompressed version. Unfortunately, I cannot just uncompress them ...
4
votes
1answer
2k views

GNU Parallel - grepping n lines for m regular expressions

The GNU parallel grepping n lines for m regular expressions example states the following: If the CPU is the limiting factor parallelization should be done on the regexps: cat regexp.txt | ...
3
votes
3answers
2k views

How can I get the first column of a 300GB txt file?

let me first explain my problem in detail. It's actually pretty simple. I have a huge .txt file, 300GB to be more preciese, and I would like to put all the distinct strings from the first column, that ...
2
votes
1answer
1k views

Best parameters for ext4 filesystem to handle virtualbox disk images

I have ext4 partition, which holds disk image files VirtualBox works with. They all are fixed-size images (i.e. files never change their size). They are defragmented as much as possible (with e4defrag)...
3
votes
1answer
414 views

Is it possible to get file list, byte range from the head of a tar+bzip2 file?

Essentially, I'm hoping someone with an advanced knowledge of tar/bz2 can answer whether this is possible. The situation is that we have a periodic 24GB feed of data from a vendor, as a .tbz file. (...
2
votes
3answers
209 views

Filesystem surprisingly large

I'm running a dual booted Windows 7 with Linux Mint right now and I just checked my filesystem size and it jumped from quite small (about 30 GB) to 200 GB recently (no clue what triggered this, I just ...
7
votes
2answers
7k views

Basic sed command on large one-line file: couldn't re-allocate memory

I have a 250 MB text file, all in one line. In this file I want to replace a characters with b characters: sed -e "s/a/b/g" < one-line-250-mb.txt It fails with: sed: couldn't re-allocate memory ...
6
votes
1answer
1k views

Merge nonzero blocks of huge (sparse) file A into huge file B

I have two partial disk images from a failing hard drive. File B contains the bulk of the disk's contents, with gaps where sector reads failed. File A is the result of telling ddrescue to retry all ...
3
votes
2answers
2k views

awk read error bad address

I have a file that is roughly 30GB. I am using, awk ' $1 <= 2000 ' myfile.txt >> myfile_new.txt But, I get this error after a while - awk: read error (Bad address) I guess this is ...
7
votes
1answer
4k views

Viewing large image on Linux

I need to view a large (50000x40000 px) png image on Linux. Unfortunately most tools (eog, convert etc.) either crashes or fail with note about too little memory. Is there a way to view this image (I ...