Skip to main content

Questions tagged [large-files]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
1 answer
52 views

Use GNU parallel with very long lines

I have a very large SQL dumpfile (30GB) that I need to edit (do some find/replace) before loading back into the database. Besides having a large size, the file also contains very long lines. Except ...
fernan's user avatar
  • 23
16 votes
5 answers
5k views

How do I read the last lines of a huge log file?

I have a log of 55GB in size. I tried: cat logfile.log | tail But this approach takes a lot of time. Is there any way to read huge files faster or any other approach?
Yi Qiang Ji's user avatar
2 votes
3 answers
924 views

EXT4 for very large (>1GB) files : increase block size, use block clusters, or both?

I'd like to format a 12 TB HDD (not SSD) with EXT4, in order to store large video files (each file being at least 1 GiB in size). I am working with an x86-64 (a.k.a. x64 or amd64) processor. There's ...
ChennyStar's user avatar
  • 1,795
1 vote
1 answer
292 views

Transferring very large dataset from cluster to a storage server

We have to move a set of very large data (in petabytes) from HPC cluster to a storage server. We have a high capacity communication link between the devices. However, the bottleneck seems to be a fast ...
Ikram Ullah's user avatar
6 votes
4 answers
988 views

Delete huge directory that causes all commands to hang

How do I delete this large directory? stat session/ File: ‘session/’ Size: 321540096 Blocks: 628040 IO Block: 4096 directory Device: 903h/2307d Inode: 11149319 Links: 2 Access: ...
Bojan Hrnkas's user avatar
9 votes
2 answers
2k views

Is rsync --append able to resume an interrupted copy process without reading all the copied data?

I need to copy one very large file (3TB) on the same machine from one external drive to another. This might take (because of low bandwidth) many days. So I want to be prepared when I have to interrupt ...
halloleo's user avatar
  • 589
3 votes
1 answer
6k views

Is cp faster than rsync during the first run?

I am currently using rsync to copy a 73GB file from a Samsung portable SSD T7 to an HPC cluster. rsync -avh path/to/dataset [email protected]:/path/to/dest The following applies: My local machine (...
mesllo's user avatar
  • 133
1 vote
3 answers
962 views

How to compare huge files with progress information

In a Unix command line context I would like to compare two truly huge files (around 1TB each), preferable with a progress indicator. I have tried diff and cmp, and they both crashed the system (macOS ...
halloleo's user avatar
  • 589
2 votes
0 answers
1k views

How to use less in a 'safe' way on files with huge lines?

I have a relatively large file, a minified json file which is about 4 Gigs. The file isn't huge...but a lot of programs choke on it because it's a single-lined file. I have noticed less works ok on it ...
David542's user avatar
  • 339
1 vote
5 answers
462 views

Pipe skip 99 lines out of every 100

I have a bash commands pipeline that produces a ton of logging text output. But mostly it repeats the previous line except for the timestamp and some minor flags, the main output data changes only ...
scythargon's user avatar
2 votes
3 answers
1k views

Remove duplicated lines containing specific text

I have a big text file containing a million lines. I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas? So the algorithm should roughly be ...
nmzik's user avatar
  • 21
2 votes
1 answer
2k views

Split large file into chunks and delete original?

I'm having trouble with a large text file (30GB) I would like to create smaller files from it (5GB lets say) But sadly I don't have no more storage (only ~10G is available). This line: split -b 5g &...
Andras Karpati's user avatar
0 votes
1 answer
457 views

How do you keep unison from syncing large files?

I am using unison file syncing software and I am aware of the config that ignores syncing files with specific regex or name. But is there a way to block syncing of large files e.g. larger than 10 MB? ...
mtk's user avatar
  • 27.8k
1 vote
0 answers
104 views

Grep something that is between two known strings

I have a large potentially zipped log file and I can identify which line number some text I'm interested in is on using: find . -name "*" -exec zgrep -C 1 -n -i -H TextToFind {} \; But in a second ...
Steven Roman's user avatar
1 vote
0 answers
157 views

I want a shared folder between Arch (host) and Android

I have a music folder of nearly 10 GB, of completely offline music. I want to keep it on my phone, so i can take it on the go. The simplest answer is a file transfer between my phone and PC, however ...
Cosmo's user avatar
  • 11
2 votes
0 answers
1k views

How to encrypt an 8TB disk with Veracrypt with hidden partition?

When I try to encrypt an USB 8tb drive (Seagate Expansion Drive) with Veracrypt to create a Hidden Veracrypt volume, I receive this error: Error: The hidden volume to be created is larger than 2 TB ...
flen's user avatar
  • 161
1 vote
1 answer
870 views

Replace text in big file using sed

I have a big file that contains a text will be replaced based on a regular expression passed to. The regular expression is tested and can find matching pattern, but when I use it with sed no text ...
H Aßdøµ's user avatar
0 votes
1 answer
600 views

Cut string on every nth line of a very large document

I have document A here: @rand1 ABCDEFBHIJKLM + <<///// @rand2 NOPQRSTUVW + <<//<<< @anotherrand XYZABCDE + <<//<<< I would like to have an output that contains ...
DNA_bash's user avatar
3 votes
2 answers
2k views

Failing to untar

I am dealing with transferring large files from one machine to another (600GB+) and I'm tarring them up using tar -cpvzf file.tar.gz -C PATH_TO_DIR DIR Once finished with the tarring process, the ...
Siewiei's user avatar
  • 33
2 votes
2 answers
2k views

How can I make my sed script run faster?

I got this script my a related question of mine - How do I insert the filename and header to the beginning of a csv find . -name '*.csv' -printf "%f\n" | sed 's/.csv$//' | xargs -I{} sed -i '1s/^/...
scientific_explorer's user avatar
1 vote
1 answer
623 views

Split large log file based on date criteria, while logging facility is running

the log file is /var/log/dovecot.info, >4GB, with date format: 2019-07-01 01:07:40 My objective is: have the /var/log/dovecot.info which starts with 2019-01-01 00:00:00 and contains up to now log ...
Miloš Đakonović's user avatar
1 vote
1 answer
66 views

Locate Large Files - Discrepancy between Parent Folder and Child Folder

System: Red Hat Enterprise Linux Server release 7.6 (Maipo), 3.10.0-957.12.1.el7.x86_64 GOAL: Find files that are large in size to see how they can be remediated. Note: Folders with "application" in ...
alcazaro's user avatar
4 votes
1 answer
2k views

What filesystem allows transferring files between Linux and OS X?

I have a file that is 12 GB that I am trying to copy from a MacBook Air to a Debian computer, using a USB. I tried formatting the USB in many different ways, such as NTFS, FAT32, OS X Journalizing but ...
Village's user avatar
  • 3,717
9 votes
1 answer
8k views

How copytruncate actually works?

we would like to understand copytruncate before rotating the file using logrotate with below configuration: /app/syslog-ng/custom/output/all_devices.log { size 200M copytruncate dateext ...
overexchange's user avatar
  • 1,546
0 votes
1 answer
2k views

Comparing two files with an easy readable GUI program?

I would like to compare two large files that are not close to 1Gb in size. Kompare crashes when the files load is large. I configured Kompare as in the photo below: I would like to present the ...
Mc Jorch's user avatar
13 votes
5 answers
33k views

How to find duplicate lines in many large files?

I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free. My goal: I want to find all all duplicate lines across ...
Lars Schneider's user avatar
133 votes
14 answers
34k views

Replace string in a huge (70GB), one line, text file

I have a huge (70GB), one line, text file and I want to replace a string (token) in it. I want to replace the token <unk>, with another dummy token (glove issue). I tried sed: sed 's/<unk&...
Christos Baziotis's user avatar
4 votes
1 answer
11k views

Why does head; tail on a large file sometimes take a long time and sometimes not?

I am writing a bash script that reads the first 10 lines and the last 10 lines of a .txt file. It looks for started (head) and completed (tail) and compares the number of occurrences using grep. The ...
Racehorse35's user avatar
3 votes
4 answers
956 views

grep: Which *patterns* are matched, not which text?

In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that. In other words, given a fairly large ...
derobert's user avatar
  • 111k
5 votes
1 answer
2k views

best way to split huge files based on a field when awk is too slow

I am having problems dealing with huge .gz files (greater than 500G). My goal is to split each of these by the 4th field within these files. There is this beautiful awk one-liner I have used before to ...
user971102's user avatar
0 votes
2 answers
2k views

Awk - Compare Value from a column with a variable, count how many times it passes for each ID

I have been trying to find a solution but now I need some help. First, as input, I have a Big File (5.5G) with this structure: scaffold4691_size302 2 T 1 scaffold4691_size302 3 ...
dilution's user avatar
4 votes
3 answers
13k views

diff two large CSV files (each 90GB) and output to another csv

This is a follow up question from Sort large CSV files (90GB), Disk quota exceeded. So now I have two CSV files sorted, as file1.csv and file2.csv each CSV file has 4 columns, e.g. file 1: ID Date ...
ChubbyRuby's user avatar
7 votes
1 answer
3k views

Sort large CSV files (90GB), Disk quota exceeded

Here is what I do right now, sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv the file is 90GB,I got this error message sort: close failed: /some_dir/sortmdWWn4: Disk quota ...
ChubbyRuby's user avatar
1 vote
2 answers
4k views

fast ways of removing beginning lines from large text file

I have a big text file (>500GB), all the ways I can find (sed/tail and others) all require write the 500GB content to disk. Is there anyway to quickly remove the first a few lines in place without ...
1a1a11a's user avatar
  • 121
1 vote
1 answer
1k views

Is there an efficient way to parse blocks of text in python?

I have an enormous file (~70GB) with lines that look like this: $ cat mybigfile.txt 5 7 1 1 0 -2 0 0 2 0 4 0 -4 0 0 4 0 0 1 -1 0 0 0 ...
Brian Fitzpatrick's user avatar
0 votes
1 answer
226 views

Using split files without merging them

I'm thinking a way of storing large files on vfat file system. Obviously, the only way to store files larger than 4GiB is to split them. I'm aware that we can use split and cat commands for splitting ...
user avatar
0 votes
1 answer
1k views

Opening big storage log file in vi

I have a Storage log in an AIX partition. When I try to edit the file with a vi command, I receive the following message: ex: 0602-101 Out of memory saving lines for undo delete lines I don't want to ...
user210129's user avatar
4 votes
1 answer
3k views

Recommended maximum percentage to fill a large ext4 data drive

The default reserved blocks percentage for ext filesystems is 5%. On a 4TB data drive this is 200GB which seems excessive to me. Obviously this can be adjusted with tune2fs: tune2fs -m <...
maloo's user avatar
  • 510
1 vote
1 answer
156 views

Word frequency gawk memory leak

I have the following bash script: #!/usr/bin/env bash grep -e '^[a-zA-Z]\{4,8\}$' data/words3.txt | tr '[:upper:]' '[:lower:]' | sort -u > data/passphrase-words.txt function wordfrequency() { ...
xenrelay's user avatar
1 vote
1 answer
750 views

Does traditional vi (not vim) have large file support?

Is there a way to compile traditional Vi (ex) with large-file (3+ GB) support? I'm trying to open a file that is around 3.5GBs in size and Vi simply says: Tmp file too large. If large-file support ...
Alexej Magura's user avatar
4 votes
3 answers
2k views

How to determine size of tar archive without creating it?

I'm archiving a few directories every night to LTO-7 tape with about 100 or so large (2GB) files in each of them. As a check that the data has been written correctly, I'm verifying that the number of ...
nippoo's user avatar
  • 161
2 votes
1 answer
1k views

Faster (uncompressed) archiving tool than tar?

I'm using tar to archive a bunch of files to LTO-7 tape. Generally each file is about 1-2GB, and there can be several hundred of them in each archive (up to ~1TB per archive). At the moment, I'm ...
nippoo's user avatar
  • 161
1 vote
2 answers
127 views

Print each line multiple but different times

I have a huge file, ~4000000 lines that look like that: file: 1 1 2 3 2 4 ... basically, each line contains a number. Considering this example I would like to generate a file that will look ...
Sagi's user avatar
  • 49
8 votes
7 answers
7k views

grep with pattern from one file (3.2Gb) matching in another file (4.8Gb)

I have two text files. One is a text file with name, email address and other fields. Some lines from file1: John:[email protected]:johnson123:22hey Erik:[email protected]:johnson133:22hey Robert:...
Axel Tobieson's user avatar
3 votes
1 answer
6k views

Transfer Size limit for SCP, SFTP, RSYNC?

Is there a maximum size limit to the size of collective size of the files that are to be transferred in one go for the above protocol. My intent is to know if for example after 1 GB any of these ...
Anurag Azad's user avatar
0 votes
3 answers
599 views

How to loop through a file and make each line a new regular expression into an awk statement?

FileA.txt: ATGCATGC GGGGGGTT TTTTT AAAA FileB.txt: asdfasdf blah2 ATGCATGC blah3 blah4 delte-me-too GGGGGGTT blah5 blah5 .... I want to compare the each line from FileA.txt and check if it is ...
Anonymous's user avatar
39 votes
3 answers
86k views

Replace text quickly in very large file

I have 25GB text file that needs a string replaced on only a few lines. I can use sed successfully but it takes a really long time to run. sed -i 's|old text|new text|g' gigantic_file.sql Is there a ...
eisaacson's user avatar
  • 491
1 vote
1 answer
2k views

"Cannot reallocate" when create file?

I am trying to create excel sheet based on multiple files on a root. I read files line by line and append in the final excel sheet. I am trying this shell script on small files and it worked 100%, ...
Ahmed Samir's user avatar
1 vote
1 answer
2k views

aix: large files

/opt/freeware/bin/ls //media/nfs/slack64/pubblica /opt/freeware/bin/ls: cannot open directory //media/nfs/slack64/pubblica: Value too large to be stored in data type This happen with coreutils-8.23 ...
elbarna's user avatar
  • 12.8k
0 votes
4 answers
7k views

Extract several lines from large text file [duplicate]

I have a large text file. I need to quickly pull a bunch of lines, say from #14600 to #14700, from this file, as a separate file. How it could be done?
disfated's user avatar
  • 138