Questions tagged [large-files]
The large-files tag has no usage guidance.
89
questions
1
vote
1
answer
151
views
Transferring very large dataset from cluster to a storage server
We have to move a set of very large data (in petabytes) from HPC cluster to a storage server. We have a high capacity communication link between the devices. However, the bottleneck seems to be a fast ...
- 111
6
votes
3
answers
610
views
Delete huge directory that causes all commands to hang
How do I delete this large directory?
stat session/
File: ‘session/’
Size: 321540096 Blocks: 628040 IO Block: 4096 directory
Device: 903h/2307d Inode: 11149319 Links: 2
Access: ...
- 201
9
votes
2
answers
1k
views
Is rsync --append able to resume an interrupted copy process without reading all the copied data?
I need to copy one very large file (3TB) on the same machine from one external drive to another. This might take (because of low bandwidth) many days.
So I want to be prepared when I have to interrupt ...
- 549
2
votes
1
answer
4k
views
Is cp faster than rsync during the first run?
I am currently using rsync to copy a 73GB file from a Samsung portable SSD T7 to an HPC cluster.
rsync -avh path/to/dataset [email protected]:/path/to/dest
The following applies:
My local machine (...
- 123
0
votes
3
answers
620
views
How to compare huge files with progress information
In a Unix command line context I would like to compare two truly huge files (around 1TB each), preferable with a progress indicator.
I have tried diff and cmp, and they both crashed the system (macOS ...
- 549
2
votes
0
answers
703
views
How to use less in a 'safe' way on files with huge lines?
I have a relatively large file, a minified json file which is about 4 Gigs. The file isn't huge...but a lot of programs choke on it because it's a single-lined file.
I have noticed less works ok on it ...
- 339
1
vote
5
answers
418
views
Pipe skip 99 lines out of every 100
I have a bash commands pipeline that produces a ton of logging text output. But mostly it repeats the previous line except for the timestamp and some minor flags, the main output data changes only ...
- 121
2
votes
3
answers
1k
views
Remove duplicated lines containing specific text
I have a big text file containing a million lines.
I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas?
So the algorithm should roughly be ...
- 21
2
votes
1
answer
2k
views
Split large file into chunks and delete original?
I'm having trouble with a large text file (30GB)
I would like to create smaller files from it (5GB lets say)
But sadly I don't have no more storage (only ~10G is available).
This line:
split -b 5g &...
0
votes
1
answer
368
views
How do you keep unison from syncing large files?
I am using unison file syncing software and I am aware of the config that ignores syncing files with specific regex or name. But is there a way to block syncing of large files e.g. larger than 10 MB?
...
- 27.2k
1
vote
0
answers
92
views
Grep something that is between two known strings
I have a large potentially zipped log file and I can identify which line number some text I'm interested in is on using:
find . -name "*" -exec zgrep -C 1 -n -i -H TextToFind {} \;
But in a second ...
- 11
1
vote
0
answers
132
views
I want a shared folder between Arch (host) and Android
I have a music folder of nearly 10 GB, of completely offline music. I want to keep it on my phone, so i can take it on the go. The simplest answer is a file transfer between my phone and PC, however ...
- 11
2
votes
0
answers
1k
views
How to encrypt an 8TB disk with Veracrypt with hidden partition?
When I try to encrypt an USB 8tb drive (Seagate Expansion Drive) with Veracrypt to create a Hidden Veracrypt volume, I receive this error:
Error: The hidden volume to be created is larger than 2 TB ...
- 161
1
vote
1
answer
692
views
Replace text in big file using sed
I have a big file that contains a text will be replaced based on a regular expression passed to.
The regular expression is tested and can find matching pattern, but when I use it with sed no text ...
- 111
0
votes
1
answer
584
views
Cut string on every nth line of a very large document
I have document A here:
@rand1
ABCDEFBHIJKLM
+
<</////
@rand2
NOPQRSTUVW
+
<<//<<<
@anotherrand
XYZABCDE
+
<<//<<<
I would like to have an output that contains ...
- 59
3
votes
2
answers
2k
views
Failing to untar
I am dealing with transferring large files from one machine to another (600GB+) and I'm tarring them up using
tar -cpvzf file.tar.gz -C PATH_TO_DIR DIR
Once finished with the tarring process, the ...
- 33
2
votes
2
answers
2k
views
How can I make my sed script run faster?
I got this script my a related question of mine - How do I insert the filename and header to the beginning of a csv
find . -name '*.csv' -printf "%f\n" |
sed 's/.csv$//' |
xargs -I{} sed -i '1s/^/...
1
vote
1
answer
582
views
Split large log file based on date criteria, while logging facility is running
the log file is /var/log/dovecot.info, >4GB, with date format:
2019-07-01 01:07:40
My objective is:
have the /var/log/dovecot.info which starts with 2019-01-01 00:00:00 and contains up to now log ...
- 963
1
vote
1
answer
59
views
Locate Large Files - Discrepancy between Parent Folder and Child Folder
System: Red Hat Enterprise Linux Server release 7.6 (Maipo), 3.10.0-957.12.1.el7.x86_64
GOAL: Find files that are large in size to see how they can be remediated.
Note: Folders with "application" in ...
- 35
4
votes
1
answer
2k
views
What filesystem allows transferring files between Linux and OS X?
I have a file that is 12 GB that I am trying to copy from a MacBook Air to a Debian computer, using a USB. I tried formatting the USB in many different ways, such as NTFS, FAT32, OS X Journalizing but ...
- 4,835
9
votes
1
answer
7k
views
How copytruncate actually works?
we would like to understand copytruncate before rotating the file using logrotate with below configuration:
/app/syslog-ng/custom/output/all_devices.log {
size 200M
copytruncate
dateext
...
- 1,496
0
votes
1
answer
2k
views
Comparing two files with an easy readable GUI program?
I would like to compare two large files that are not close to 1Gb in size.
Kompare crashes when the files load is large. I configured Kompare as in the photo below:
I would like to present the ...
- 3
13
votes
5
answers
32k
views
How to find duplicate lines in many large files?
I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free.
My goal: I want to find all all duplicate lines across ...
- 242
133
votes
14
answers
33k
views
Replace string in a huge (70GB), one line, text file
I have a huge (70GB), one line, text file and I want to replace a string (token) in it.
I want to replace the token <unk>, with another dummy token (glove issue).
I tried sed:
sed 's/<unk&...
- 1,457
4
votes
1
answer
10k
views
Why does head; tail on a large file sometimes take a long time and sometimes not?
I am writing a bash script that reads the first 10 lines and the last 10 lines of a .txt file. It looks for started (head) and completed (tail) and compares the number of occurrences using grep. The ...
- 53
3
votes
4
answers
932
views
grep: Which *patterns* are matched, not which text?
In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that.
In other words, given a fairly large ...
- 108k
5
votes
1
answer
2k
views
best way to split huge files based on a field when awk is too slow
I am having problems dealing with huge .gz files (greater than 500G). My goal is to split each of these by the 4th field within these files. There is this beautiful awk one-liner I have used before to ...
- 175
0
votes
2
answers
1k
views
Awk - Compare Value from a column with a variable, count how many times it passes for each ID
I have been trying to find a solution but now I need some help.
First, as input, I have a Big File (5.5G) with this structure:
scaffold4691_size302 2 T 1
scaffold4691_size302 3 ...
- 1
4
votes
3
answers
13k
views
diff two large CSV files (each 90GB) and output to another csv
This is a follow up question from Sort large CSV files (90GB), Disk quota exceeded.
So now I have two CSV files sorted, as file1.csv and file2.csv
each CSV file has 4 columns, e.g.
file 1:
ID Date ...
- 123
7
votes
1
answer
3k
views
Sort large CSV files (90GB), Disk quota exceeded
Here is what I do right now,
sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv
the file is 90GB,I got this error message
sort: close failed: /some_dir/sortmdWWn4: Disk quota ...
- 123
1
vote
2
answers
4k
views
fast ways of removing beginning lines from large text file
I have a big text file (>500GB), all the ways I can find (sed/tail and others) all require write the 500GB content to disk. Is there anyway to quickly remove the first a few lines in place without ...
- 121
1
vote
1
answer
1k
views
Is there an efficient way to parse blocks of text in python?
I have an enormous file (~70GB) with lines that look like this:
$ cat mybigfile.txt
5 7
1 1 0 -2 0 0 2
0 4 0 -4 0 0 4
0 0 1 -1 0 0 0
...
- 2,785
0
votes
1
answer
200
views
Using split files without merging them
I'm thinking a way of storing large files on vfat file system. Obviously, the only way to store files larger than 4GiB is to split them. I'm aware that we can use split and cat commands for splitting ...
user210007
0
votes
1
answer
1k
views
Opening big storage log file in vi
I have a Storage log in an AIX partition. When I try to edit the file with a vi command, I receive the following message: ex: 0602-101 Out of memory saving lines for undo delete lines
I don't want to ...
- 27
4
votes
1
answer
2k
views
Recommended maximum percentage to fill a large ext4 data drive
The default reserved blocks percentage for ext filesystems is 5%. On a 4TB data drive this is 200GB which seems excessive to me.
Obviously this can be adjusted with tune2fs:
tune2fs -m <...
- 510
1
vote
1
answer
151
views
Word frequency gawk memory leak
I have the following bash script:
#!/usr/bin/env bash
grep -e '^[a-zA-Z]\{4,8\}$' data/words3.txt | tr '[:upper:]' '[:lower:]' | sort -u > data/passphrase-words.txt
function wordfrequency() {
...
- 11
1
vote
1
answer
733
views
Does traditional vi (not vim) have large file support?
Is there a way to compile traditional Vi (ex) with large-file (3+ GB) support?
I'm trying to open a file that is around 3.5GBs in size and Vi simply says: Tmp file too large. If large-file support ...
- 4,396
3
votes
2
answers
2k
views
How to determine size of tar archive without creating it?
I'm archiving a few directories every night to LTO-7 tape with about 100 or so large (2GB) files in each of them.
As a check that the data has been written correctly, I'm verifying that the number of ...
- 151
2
votes
1
answer
960
views
Faster (uncompressed) archiving tool than tar?
I'm using tar to archive a bunch of files to LTO-7 tape. Generally each file is about 1-2GB, and there can be several hundred of them in each archive (up to ~1TB per archive).
At the moment, I'm ...
- 151
1
vote
2
answers
127
views
Print each line multiple but different times
I have a huge file, ~4000000 lines that look like that:
file:
1
1
2
3
2
4
...
basically, each line contains a number.
Considering this example I would like to generate a file that will look ...
- 49
8
votes
7
answers
6k
views
grep with pattern from one file (3.2Gb) matching in another file (4.8Gb)
I have two text files. One is a text file with name, email address and other fields. Some lines from file1:
John:[email protected]:johnson123:22hey
Erik:[email protected]:johnson133:22hey
Robert:...
- 149
3
votes
1
answer
5k
views
Transfer Size limit for SCP, SFTP, RSYNC?
Is there a maximum size limit to the size of collective size of the files that are to be transferred in one go for the above protocol. My intent is to know if for example after 1 GB any of these ...
- 31
0
votes
3
answers
581
views
How to loop through a file and make each line a new regular expression into an awk statement?
FileA.txt:
ATGCATGC
GGGGGGTT
TTTTT
AAAA
FileB.txt:
asdfasdf
blah2
ATGCATGC
blah3
blah4
delte-me-too
GGGGGGTT
blah5
blah5
....
I want to compare the each line from FileA.txt and check if it is ...
- 33
38
votes
3
answers
83k
views
Replace text quickly in very large file
I have 25GB text file that needs a string replaced on only a few lines. I can use sed successfully but it takes a really long time to run.
sed -i 's|old text|new text|g' gigantic_file.sql
Is there a ...
- 481
1
vote
1
answer
1k
views
"Cannot reallocate" when create file?
I am trying to create excel sheet based on multiple files on a root. I read files line by line and append in the final excel sheet.
I am trying this shell script on small files and it worked 100%, ...
- 23
1
vote
1
answer
2k
views
aix: large files
/opt/freeware/bin/ls //media/nfs/slack64/pubblica
/opt/freeware/bin/ls: cannot open directory //media/nfs/slack64/pubblica: Value too large to be stored in data type
This happen with coreutils-8.23 ...
- 12.4k
0
votes
4
answers
7k
views
Extract several lines from large text file [duplicate]
I have a large text file.
I need to quickly pull a bunch of lines, say from #14600 to #14700, from this file, as a separate file.
How it could be done?
- 138
2
votes
4
answers
12k
views
Fast way to add/append to a large file [closed]
I have a bash script that reads a rather large file, line by line, and for each line does some processing and writes the results to another file. Currently I am using echo to append to the end of the ...
18
votes
2
answers
3k
views
Why are these files in an ext4 volume fragmented?
I have a 900GB ext4 partition on a (magnetic) hard drive that has no defects and no bad sectors. The partition is completely empty except for an empty lost+found directory. The partition was formatted ...
- 4,047
8
votes
4
answers
8k
views
how to create multi tar archives for a huge folder
I have a large folder with 30M small files. I hope to backup the folder into 30 archives, each tar.gz file will have 1M files. The reason to split into multi archives is that to untar one single large ...
- 283