Questions tagged [large-files]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
45 views

Grep something that is between two known strings

I have a large potentially zipped log file and I can identify which line number some text I'm interested in is on using: find . -name "*" -exec zgrep -C 1 -n -i -H TextToFind {} \; But in a second ...
1
vote
0answers
14 views

I want a shared folder between Arch (host) and Android

I have a music folder of nearly 10 GB, of completely offline music. I want to keep it on my phone, so i can take it on the go. The simplest answer is a file transfer between my phone and PC, however ...
1
vote
0answers
114 views

How to encrypt an 8TB disk with Veracrypt with hidden partition?

When I try to encrypt an USB 8tb drive (Seagate Expansion Drive) with Veracrypt to create a Hidden Veracrypt volume, I receive this error: Error: The hidden volume to be created is larger than 2 TB ...
1
vote
1answer
60 views

Replace text in big file using sed

I have a big file that contains a text will be replaced based on a regular expression passed to. The regular expression is tested and can find matching pattern, but when I use it with sed no text ...
0
votes
1answer
76 views

Cut string on every nth line of a very large document

I have document A here: @rand1 ABCDEFBHIJKLM + <<///// @rand2 NOPQRSTUVW + <<//<<< @anotherrand XYZABCDE + <<//<<< I would like to have an output that contains ...
3
votes
2answers
132 views

Failing to untar

I am dealing with transferring large files from one machine to another (600GB+) and I'm tarring them up using tar -cpvzf file.tar.gz -C PATH_TO_DIR DIR Once finished with the tarring process, the ...
2
votes
2answers
121 views

How can I make my sed script run faster?

I got this script my a related question of mine - How do I insert the filename and header to the beginning of a csv find . -name '*.csv' -printf "%f\n" | sed 's/.csv$//' | xargs -I{} sed -i '1s/^/...
0
votes
1answer
71 views

Split large log file based on date criteria, while logging facility is running

the log file is /var/log/dovecot.info, >4GB, with date format: 2019-07-01 01:07:40 My objective is: have the /var/log/dovecot.info which starts with 2019-01-01 00:00:00 and contains up to now log ...
1
vote
1answer
32 views

Locate Large Files - Discrepancy between Parent Folder and Child Folder

System: Red Hat Enterprise Linux Server release 7.6 (Maipo), 3.10.0-957.12.1.el7.x86_64 GOAL: Find files that are large in size to see how they can be remediated. Note: Folders with "application" in ...
4
votes
4answers
665 views

Fast way to extract lines from a large file with 80 billion lines

I have a large file with 80 billion lines. Now I want to extract a few lines (around 10000) which I know the line number, what is the fastest way to deal with it. Is it possible to extract those ...
3
votes
1answer
507 views

What filesystem allows transferring files between Linux and OS X?

I have a file that is 12 GB that I am trying to copy from a MacBook Air to a Debian computer, using a USB. I tried formatting the USB in many different ways, such as NTFS, FAT32, OS X Journalizing but ...
4
votes
1answer
2k views

How copytruncate actually works? [closed]

we would like to understand copytruncate before rotating the file using logrotate with below configuration: /app/syslog-ng/custom/output/all_devices.log { size 200M copytruncate dateext ...
0
votes
1answer
378 views

Comparing two files with an easy readable GUI program?

I would like to compare two large files that are not close to 1Gb in size. Kompare crashes when the files load is large. I configured Kompare as in the photo below: I would like to present the ...
8
votes
4answers
13k views

How to find duplicate lines in many large files?

I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free. My goal: I want to find all all duplicate lines across ...
127
votes
14answers
25k views

Replace string in a huge (70GB), one line, text file

I have a huge (70GB), one line, text file and I want to replace a string (token) in it. I want to replace the token <unk>, with another dummy token (glove issue). I tried sed: sed 's/<unk&...
2
votes
1answer
3k views

Why does head; tail on a large file sometimes take a long time and sometimes not?

I am writing a bash script that reads the first 10 lines and the last 10 lines of a .txt file. It looks for started (head) and completed (tail) and compares the number of occurrences using grep. The ...
3
votes
4answers
454 views

grep: Which *patterns* are matched, not which text?

In maintaining our local logcheck rules, it'd be nice to know which ones are no longer being used. I'm trying to find a reasonably efficient way of doing that. In other words, given a fairly large ...
5
votes
1answer
707 views

best way to split huge files based on a field when awk is too slow

I am having problems dealing with huge .gz files (greater than 500G). My goal is to split each of these by the 4th field within these files. There is this beautiful awk one-liner I have used before to ...
0
votes
2answers
574 views

Awk - Compare Value from a column with a variable, count how many times it passes for each ID

I have been trying to find a solution but now I need some help. First, as input, I have a Big File (5.5G) with this structure: scaffold4691_size302 2 T 1 scaffold4691_size302 3 ...
3
votes
2answers
7k views

diff two large CSV files (each 90GB) and output to another csv

This is a follow up question from Sort large CSV files (90GB), Disk quota exceeded. So now I have two CSV files sorted, as file1.csv and file2.csv each CSV file has 4 columns, e.g. file 1: ID Date ...
7
votes
1answer
1k views

Sort large CSV files (90GB), Disk quota exceeded

Here is what I do right now, sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv the file is 90GB,I got this error message sort: close failed: /some_dir/sortmdWWn4: Disk quota ...
1
vote
2answers
2k views

fast ways of removing beginning lines from large text file

I have a big text file (>500GB), all the ways I can find (sed/tail and others) all require write the 500GB content to disk. Is there anyway to quickly remove the first a few lines in place without ...
1
vote
1answer
546 views

Is there an efficient way to parse blocks of text in python?

I have an enormous file (~70GB) with lines that look like this: $ cat mybigfile.txt 5 7 1 1 0 -2 0 0 2 0 4 0 -4 0 0 4 0 0 1 -1 0 0 0 ...
0
votes
1answer
61 views

Using split files without merging them

I'm thinking a way of storing large files on vfat file system. Obviously, the only way to store files larger than 4GiB is to split them. I'm aware that we can use split and cat commands for splitting ...
0
votes
1answer
769 views

Opening big storage log file in vi

I have a Storage log in an AIX partition. When I try to edit the file with a vi command, I receive the following message: ex: 0602-101 Out of memory saving lines for undo delete lines I don't want to ...
4
votes
1answer
747 views

Recommended maximum percentage to fill a large ext4 data drive

The default reserved blocks percentage for ext filesystems is 5%. On a 4TB data drive this is 200GB which seems excessive to me. Obviously this can be adjusted with tune2fs: tune2fs -m <...
1
vote
1answer
86 views

Word frequency gawk memory leak

I have the following bash script: #!/usr/bin/env bash grep -e '^[a-zA-Z]\{4,8\}$' data/words3.txt | tr '[:upper:]' '[:lower:]' | sort -u > data/passphrase-words.txt function wordfrequency() { ...
1
vote
1answer
401 views

Does traditional vi (not vim) have large file support?

Is there a way to compile traditional Vi (ex) with large-file (3+ GB) support? I'm trying to open a file that is around 3.5GBs in size and Vi simply says: Tmp file too large. If large-file support ...
3
votes
2answers
1k views

How to determine size of tar archive without creating it?

I'm archiving a few directories every night to LTO-7 tape with about 100 or so large (2GB) files in each of them. As a check that the data has been written correctly, I'm verifying that the number of ...
2
votes
1answer
481 views

Faster (uncompressed) archiving tool than tar?

I'm using tar to archive a bunch of files to LTO-7 tape. Generally each file is about 1-2GB, and there can be several hundred of them in each archive (up to ~1TB per archive). At the moment, I'm ...
1
vote
2answers
120 views

Print each line multiple but different times

I have a huge file, ~4000000 lines that look like that: file: 1 1 2 3 2 4 ... basically, each line contains a number. Considering this example I would like to generate a file that will look ...
7
votes
7answers
3k views

grep with pattern from one file (3.2Gb) matching in another file (4.8Gb)

I have two text files. One is a text file with name, email address and other fields. Some lines from file1: John:myemail@gmail.com:johnson123:22hey Erik:thatwhatsup@gmail.com:johnson133:22hey Robert:...
3
votes
1answer
3k views

Transfer Size limit for SCP, SFTP, RSYNC?

Is there a maximum size limit to the size of collective size of the files that are to be transferred in one go for the above protocol. My intent is to know if for example after 1 GB any of these ...
0
votes
3answers
292 views

How to loop through a file and make each line a new regular expression into an awk statement?

FileA.txt: ATGCATGC GGGGGGTT TTTTT AAAA FileB.txt: asdfasdf blah2 ATGCATGC blah3 blah4 delte-me-too GGGGGGTT blah5 blah5 .... I want to compare the each line from FileA.txt and check if it is ...
25
votes
3answers
47k views

Replace text quickly in very large file

I have 25GB text file that needs a string replaced on only a few lines. I can use sed successfully but it takes a really long time to run. sed -i 's|old text|new text|g' gigantic_file.sql Is there a ...
1
vote
1answer
828 views

“Cannot reallocate” when create file?

I am trying to create excel sheet based on multiple files on a root. I read files line by line and append in the final excel sheet. I am trying this shell script on small files and it worked 100%, ...
1
vote
1answer
2k views

aix: large files

/opt/freeware/bin/ls //media/nfs/slack64/pubblica /opt/freeware/bin/ls: cannot open directory //media/nfs/slack64/pubblica: Value too large to be stored in data type This happen with coreutils-8.23 ...
-2
votes
4answers
2k views

Extract several lines from large text file [duplicate]

I have a large text file. I need to quickly pull a bunch of lines, say from #14600 to #14700, from this file, as a separate file. How it could be done?
2
votes
4answers
7k views

Fast way to add/append to a large file [closed]

I have a bash script that reads a rather large file, line by line, and for each line does some processing and writes the results to another file. Currently I am using echo to append to the end of the ...
19
votes
2answers
2k views

Why are these files in an ext4 volume fragmented?

I have a 900GB ext4 partition on a (magnetic) hard drive that has no defects and no bad sectors. The partition is completely empty except for an empty lost+found directory. The partition was formatted ...
5
votes
3answers
4k views

how to create multi tar archives for a huge folder

I have a large folder with 30M small files. I hope to backup the folder into 30 archives, each tar.gz file will have 1M files. The reason to split into multi archives is that to untar one single large ...
4
votes
3answers
1k views

deduplication of lines in a large file

The size of the file is 962,120,335 bytes. HP-UX ******B.11.31 U ia64 ****** unlimited-user license hostname> what /usr/bin/awk /usr/bin/awk: main.c $Date: 2009/02/17 15:25:17 $Revision: ...
27
votes
3answers
48k views

Transferring large (8 GB) files over ssh

I tried it with SCP, but it says "Negative file size". >scp matlab.iso xxx@xxx:/matlab.iso matlab.iso: Negative file size Also tried using SFTP, worked fine until 2 GB of the file had transferred,...
2
votes
2answers
79 views

Create small record files from a big group of record file

I have a file like this : Mainfile: Record|1111|ABC text in between for record 1 text in between for record 1 Record|2222|XYZ text in between for record 2 Record|3333|XYZ ...
1
vote
2answers
1k views

Read file and find all occurrences and generate hash for the content between the occurrences

I have a file like this : 1 Record|1111|ABC 2 text in between for record 1 3 text in between for record 1 4 Record|2222|XYZ 5 text in between for record 2 6 Record|3333|XYZ 7 text ...
2
votes
1answer
2k views

How to handle repetitive download of some specific large files that come from the internet

We are going to offer an App which will allow our customers to download some sort of audio guide for the castle I work for. I would really like to not have every user who downloads the App (through ...
1
vote
1answer
766 views

Server reports incorrect file size when downloading a large file

I have a large tarball file on an OS X web server that I uploaded via sshfs. The file on my computer prior to uploading is 5345834658 bytes. The file after copying is 5345834658 bytes on the server so ...
8
votes
3answers
2k views

How to recompress 2 million gzip files without storing them twice?

I have about 2 million (60GiB) of gzipped small files and I would like to create a compressed archive containing all of them in an uncompressed version. Unfortunately, I cannot just uncompress them ...
4
votes
1answer
2k views

GNU Parallel - grepping n lines for m regular expressions

The GNU parallel grepping n lines for m regular expressions example states the following: If the CPU is the limiting factor parallelization should be done on the regexps: cat regexp.txt | ...
3
votes
3answers
2k views

How can I get the first column of a 300GB txt file?

let me first explain my problem in detail. It's actually pretty simple. I have a huge .txt file, 300GB to be more preciese, and I would like to put all the distinct strings from the first column, that ...