Questions tagged [large-files]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
100 views

Is cp faster than rsync during the first run?

I am currently using rsync to copy a 73GB file from a Samsung portable SSD T7 to an HPC cluster. rsync -avh path/to/dataset me@server.somewhere:/path/to/dest The following applies: My local machine (...
user avatar
  • 113
0 votes
3 answers
74 views

How to compare huge files with progress information

In a Unix command line context I would like to compare two truly huge files (around 1TB each), preferable with a progress indicator. I have tried diff and cmp, and they both crashed the system (macOS ...
user avatar
  • 335
1 vote
0 answers
130 views

How to use less in a 'safe' way on files with huge lines?

I have a relatively large file, a minified json file which is about 4 Gigs. The file isn't huge...but a lot of programs choke on it because it's a single-lined file. I have noticed less works ok on it ...
user avatar
  • 319
1 vote
5 answers
224 views

Pipe skip 99 lines out of every 100

I have a bash commands pipeline that produces a ton of logging text output. But mostly it repeats the previous line except for the timestamp and some minor flags, the main output data changes only ...
user avatar
1 vote
3 answers
253 views

Remove duplicated lines containing specific text

I have a big text file containing a million lines. I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas? So the algorithm should roughly be ...
user avatar
  • 11
2 votes
1 answer
643 views

Split large file into chunks and delete original?

I'm having trouble with a large text file (30GB) I would like to create smaller files from it (5GB lets say) But sadly I don't have no more storage (only ~10G is available). This line: split -b 5g &...
user avatar
0 votes
1 answer
144 views

How do you keep unison from syncing large files?

I am using unison file syncing software and I am aware of the config that ignores syncing files with specific regex or name. But is there a way to block syncing of large files e.g. larger than 10 MB? ...
user avatar
  • 24.4k
1 vote
0 answers
64 views

Grep something that is between two known strings

I have a large potentially zipped log file and I can identify which line number some text I'm interested in is on using: find . -name "*" -exec zgrep -C 1 -n -i -H TextToFind {} \; But in a second ...
user avatar
1 vote
0 answers
83 views

I want a shared folder between Arch (host) and Android

I have a music folder of nearly 10 GB, of completely offline music. I want to keep it on my phone, so i can take it on the go. The simplest answer is a file transfer between my phone and PC, however ...
user avatar
  • 11
1 vote
0 answers
671 views

How to encrypt an 8TB disk with Veracrypt with hidden partition?

When I try to encrypt an USB 8tb drive (Seagate Expansion Drive) with Veracrypt to create a Hidden Veracrypt volume, I receive this error: Error: The hidden volume to be created is larger than 2 TB ...
user avatar
  • 121
1 vote
1 answer
191 views

Replace text in big file using sed

I have a big file that contains a text will be replaced based on a regular expression passed to. The regular expression is tested and can find matching pattern, but when I use it with sed no text ...
user avatar
0 votes
1 answer
343 views

Cut string on every nth line of a very large document

I have document A here: @rand1 ABCDEFBHIJKLM + <<///// @rand2 NOPQRSTUVW + <<//<<< @anotherrand XYZABCDE + <<//<<< I would like to have an output that contains ...
user avatar
3 votes
2 answers
1k views

Failing to untar

I am dealing with transferring large files from one machine to another (600GB+) and I'm tarring them up using tar -cpvzf file.tar.gz -C PATH_TO_DIR DIR Once finished with the tarring process, the ...
user avatar
  • 33
2 votes
2 answers
615 views

How can I make my sed script run faster?

I got this script my a related question of mine - How do I insert the filename and header to the beginning of a csv find . -name '*.csv' -printf "%f\n" | sed 's/.csv$//' | xargs -I{} sed -i '1s/^/...
user avatar
1 vote
1 answer
351 views

Split large log file based on date criteria, while logging facility is running

the log file is /var/log/dovecot.info, >4GB, with date format: 2019-07-01 01:07:40 My objective is: have the /var/log/dovecot.info which starts with 2019-01-01 00:00:00 and contains up to now log ...
user avatar
1 vote
1 answer
37 views

Locate Large Files - Discrepancy between Parent Folder and Child Folder

System: Red Hat Enterprise Linux Server release 7.6 (Maipo), 3.10.0-957.12.1.el7.x86_64 GOAL: Find files that are large in size to see how they can be remediated. Note: Folders with "application" in ...
user avatar
3 votes
1 answer
1k views

What filesystem allows transferring files between Linux and OS X?

I have a file that is 12 GB that I am trying to copy from a MacBook Air to a Debian computer, using a USB. I tried formatting the USB in many different ways, such as NTFS, FAT32, OS X Journalizing but ...
user avatar
  • 5,437
9 votes
1 answer
5k views

How copytruncate actually works?

we would like to understand copytruncate before rotating the file using logrotate with below configuration: /app/syslog-ng/custom/output/all_devices.log { size 200M copytruncate dateext ...
user avatar
  • 1,286
0 votes
1 answer
1k views

Comparing two files with an easy readable GUI program?

I would like to compare two large files that are not close to 1Gb in size. Kompare crashes when the files load is large. I configured Kompare as in the photo below: I would like to present the ...
user avatar
10 votes
5 answers
27k views

How to find duplicate lines in many large files?

I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free. My goal: I want to find all all duplicate lines across ...
user avatar
132 votes
14 answers
31k views

Replace string in a huge (70GB), one line, text file

I have a huge (70GB), one line, text file and I want to replace a string (token) in it. I want to replace the token <unk>, with another dummy token (glove issue). I tried sed: sed 's/<unk&...
user avatar
4 votes
1 answer
8k views

Why does head; tail on a large file sometimes take a long time and sometimes not?

I am writing a bash script that reads the first 10 lines and the last 10 lines of a .txt file. It looks for started (head) and completed (tail) and compares the number of occurrences using grep. The ...
user avatar
3 votes
4 answers
757 views

grep: Which *patterns* are matched, not which text?

In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that. In other words, given a fairly large ...
user avatar
  • 102k
5 votes
1 answer
1k views

best way to split huge files based on a field when awk is too slow

I am having problems dealing with huge .gz files (greater than 500G). My goal is to split each of these by the 4th field within these files. There is this beautiful awk one-liner I have used before to ...
user avatar
0 votes
2 answers
1k views

Awk - Compare Value from a column with a variable, count how many times it passes for each ID

I have been trying to find a solution but now I need some help. First, as input, I have a Big File (5.5G) with this structure: scaffold4691_size302 2 T 1 scaffold4691_size302 3 ...
user avatar
3 votes
3 answers
11k views

diff two large CSV files (each 90GB) and output to another csv

This is a follow up question from Sort large CSV files (90GB), Disk quota exceeded. So now I have two CSV files sorted, as file1.csv and file2.csv each CSV file has 4 columns, e.g. file 1: ID Date ...
user avatar
7 votes
1 answer
2k views

Sort large CSV files (90GB), Disk quota exceeded

Here is what I do right now, sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv the file is 90GB,I got this error message sort: close failed: /some_dir/sortmdWWn4: Disk quota ...
user avatar
1 vote
2 answers
3k views

fast ways of removing beginning lines from large text file

I have a big text file (>500GB), all the ways I can find (sed/tail and others) all require write the 500GB content to disk. Is there anyway to quickly remove the first a few lines in place without ...
user avatar
  • 121
1 vote
1 answer
1k views

Is there an efficient way to parse blocks of text in python?

I have an enormous file (~70GB) with lines that look like this: $ cat mybigfile.txt 5 7 1 1 0 -2 0 0 2 0 4 0 -4 0 0 4 0 0 1 -1 0 0 0 ...
user avatar
0 votes
1 answer
139 views

Using split files without merging them

I'm thinking a way of storing large files on vfat file system. Obviously, the only way to store files larger than 4GiB is to split them. I'm aware that we can use split and cat commands for splitting ...
user avatar
0 votes
1 answer
1k views

Opening big storage log file in vi

I have a Storage log in an AIX partition. When I try to edit the file with a vi command, I receive the following message: ex: 0602-101 Out of memory saving lines for undo delete lines I don't want to ...
user avatar
  • 27
4 votes
1 answer
2k views

Recommended maximum percentage to fill a large ext4 data drive

The default reserved blocks percentage for ext filesystems is 5%. On a 4TB data drive this is 200GB which seems excessive to me. Obviously this can be adjusted with tune2fs: tune2fs -m <...
user avatar
  • 430
1 vote
1 answer
126 views

Word frequency gawk memory leak

I have the following bash script: #!/usr/bin/env bash grep -e '^[a-zA-Z]\{4,8\}$' data/words3.txt | tr '[:upper:]' '[:lower:]' | sort -u > data/passphrase-words.txt function wordfrequency() { ...
user avatar
1 vote
1 answer
676 views

Does traditional vi (not vim) have large file support?

Is there a way to compile traditional Vi (ex) with large-file (3+ GB) support? I'm trying to open a file that is around 3.5GBs in size and Vi simply says: Tmp file too large. If large-file support ...
user avatar
3 votes
2 answers
2k views

How to determine size of tar archive without creating it?

I'm archiving a few directories every night to LTO-7 tape with about 100 or so large (2GB) files in each of them. As a check that the data has been written correctly, I'm verifying that the number of ...
user avatar
  • 151
2 votes
1 answer
727 views

Faster (uncompressed) archiving tool than tar?

I'm using tar to archive a bunch of files to LTO-7 tape. Generally each file is about 1-2GB, and there can be several hundred of them in each archive (up to ~1TB per archive). At the moment, I'm ...
user avatar
  • 151
1 vote
2 answers
126 views

Print each line multiple but different times

I have a huge file, ~4000000 lines that look like that: file: 1 1 2 3 2 4 ... basically, each line contains a number. Considering this example I would like to generate a file that will look ...
user avatar
  • 49
8 votes
7 answers
5k views

grep with pattern from one file (3.2Gb) matching in another file (4.8Gb)

I have two text files. One is a text file with name, email address and other fields. Some lines from file1: John:myemail@gmail.com:johnson123:22hey Erik:thatwhatsup@gmail.com:johnson133:22hey Robert:...
user avatar
3 votes
1 answer
4k views

Transfer Size limit for SCP, SFTP, RSYNC?

Is there a maximum size limit to the size of collective size of the files that are to be transferred in one go for the above protocol. My intent is to know if for example after 1 GB any of these ...
user avatar
0 votes
3 answers
533 views

How to loop through a file and make each line a new regular expression into an awk statement?

FileA.txt: ATGCATGC GGGGGGTT TTTTT AAAA FileB.txt: asdfasdf blah2 ATGCATGC blah3 blah4 delte-me-too GGGGGGTT blah5 blah5 .... I want to compare the each line from FileA.txt and check if it is ...
user avatar
34 votes
3 answers
72k views

Replace text quickly in very large file

I have 25GB text file that needs a string replaced on only a few lines. I can use sed successfully but it takes a really long time to run. sed -i 's|old text|new text|g' gigantic_file.sql Is there a ...
user avatar
  • 441
1 vote
1 answer
1k views

"Cannot reallocate" when create file?

I am trying to create excel sheet based on multiple files on a root. I read files line by line and append in the final excel sheet. I am trying this shell script on small files and it worked 100%, ...
user avatar
1 vote
1 answer
2k views

aix: large files

/opt/freeware/bin/ls //media/nfs/slack64/pubblica /opt/freeware/bin/ls: cannot open directory //media/nfs/slack64/pubblica: Value too large to be stored in data type This happen with coreutils-8.23 ...
user avatar
  • 11k
-1 votes
4 answers
5k views

Extract several lines from large text file [duplicate]

I have a large text file. I need to quickly pull a bunch of lines, say from #14600 to #14700, from this file, as a separate file. How it could be done?
user avatar
  • 109
2 votes
4 answers
10k views

Fast way to add/append to a large file [closed]

I have a bash script that reads a rather large file, line by line, and for each line does some processing and writes the results to another file. Currently I am using echo to append to the end of the ...
user avatar
18 votes
2 answers
3k views

Why are these files in an ext4 volume fragmented?

I have a 900GB ext4 partition on a (magnetic) hard drive that has no defects and no bad sectors. The partition is completely empty except for an empty lost+found directory. The partition was formatted ...
user avatar
  • 3,583
8 votes
4 answers
7k views

how to create multi tar archives for a huge folder

I have a large folder with 30M small files. I hope to backup the folder into 30 archives, each tar.gz file will have 1M files. The reason to split into multi archives is that to untar one single large ...
user avatar
  • 283
5 votes
3 answers
2k views

deduplication of lines in a large file

The size of the file is 962,120,335 bytes. HP-UX ******B.11.31 U ia64 ****** unlimited-user license hostname> what /usr/bin/awk /usr/bin/awk: main.c $Date: 2009/02/17 15:25:17 $Revision: ...
user avatar
31 votes
3 answers
74k views

Transferring large (8 GB) files over ssh

I tried it with SCP, but it says "Negative file size". >scp matlab.iso xxx@xxx:/matlab.iso matlab.iso: Negative file size Also tried using SFTP, worked fine until 2 GB of the file had transferred,...
user avatar
  • 513
2 votes
2 answers
93 views

Create small record files from a big group of record file

I have a file like this : Mainfile: Record|1111|ABC text in between for record 1 text in between for record 1 Record|2222|XYZ text in between for record 2 Record|3333|XYZ ...
user avatar