The large-files tag has no wiki summary.
30
votes
3answers
9k views
How to remove duplicate lines inside a text file?
A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table).
What I need is to remove all the repetitions ...
18
votes
6answers
2k views
Is there a way to modify a file in-place?
I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
12
votes
5answers
2k views
How can I edit a large file in place?
I have a few files sized > 1 GB each. I need to remove last few bytes from the files. How can I do it? I prefer to edit file in place to save disk space.
I am on HP-UX.
9
votes
2answers
3k views
Number of files per directory
I have a directory with about 100000 small files (each file is from 1-3 lines, each file is a text file). In size the directory isn't very big (< 2GB). This data lives in a professionally ...
6
votes
4answers
3k views
cat line X to line Y on a huge file
Say I have a huge text file (>2GB) and I just want to cat the lines X to Y (e.g. 57890000 to 57890010).
From what I understand I can do this by piping head into tail or viceversa, i.e.
head -A ...
5
votes
1answer
999 views
largefile feature at creating file-system
Is useful to use -T largefile flag at creating a file-system for a partition with big files like video, and audio in flac format?
I tested the same partition with that flag and without it, and using ...
5
votes
2answers
129 views
Emacs: Open a buffer with all lines between lines X to Y from a huge file
In the same spirit as this other question: cat line X to line Y on a huge file:
Is there a way to open from within Emacs (and
show on a buffer) a given set of lines (e.g. all lines between line X ...
5
votes
1answer
253 views
What linux editor can open a 200G file for editing within a minute or two?
I'm looking for an editor that will open the file in chunks (not try to read the whole file into memory) as I'm trying to hand-edit a 200G file.
4
votes
2answers
2k views
Diffing two big text files
I have two big files (6GB each). They are unsorted, with linefeeds (\n) as separators. How can I diff them? It should take under 24h.
4
votes
2answers
75 views
Out of memory while using sed with multiline expressions on giant file
I am currently trying to remove all newlines that are not preceded by a closing parenthesis, so I came up with this expression:
sed -r -i -e ":a;N;$!ba;s/([^\)])\n/\1/g;d" reallyBigFile.log
It does ...
4
votes
1answer
348 views
Behavior of mmap'd memory on memory pressure
I have a large tar file (60GB) containing image files. I'm using mmap() on this entire file to read in these images, which are accessed randomly.
I'm using mmap() for the following reasons:
Thread ...
3
votes
1answer
483 views
Can I safely delete ~/Trash file?
My Uni account is over disk quota and requesting more quota takes time. Unfortunately, I need disk space now and I noticed that the Trash file in my home directory is quite a large file.
Now I’m a ...
3
votes
1answer
87 views
ed: set first line as the default current line
I am trying to use ed to edit the first line of a large file, but I have to wait for ed to read all lines. Is there any way I could stop ed from reading the whole file and start editing immediately, ...
2
votes
2answers
565 views
Easy way to copy lines from one file to another [duplicate]
Possible Duplicate:
cat line X to line Y on a huge file
Very simple issue but can't seem to find a simple resolution!
I have a massive text file from which I only need around 150 lines. ...
2
votes
1answer
1k views
Edit really big files with vi
How can I edit a really big file with vi? e.g.: a 20 GByte log file. Are there any modifications for it to handle this size?
2
votes
2answers
576 views
how to find offset of one binary file inside another?
I have two binary files.
One of few hundreds kilos and other of few gigabytes.
I want to know whether the whole, smaller, file is contained within the larger one and if so then what is the offset from ...
2
votes
3answers
327 views
Uncopyable 5GB file, BTRFS
I have a 5.3GB tar file, located on a BTRFS partitioned eSATA drive. When trying to copy the file to another disk, or expand the tar the process seems to freeze. No errors are generated, and waiting ...
1
vote
4answers
670 views
Grepping over a huge file performance
I have FILE_A which has over 300K lines and FILE_B which has over 30M lines.
I created a bash script that greps each line in FILE_A over in FILE_B and writes the result of the grep to a new file.
...
1
vote
2answers
174 views
GUI File editor with large file support
Can anyone recommend a file editor, preferably free, that handles large files as well as EmEditor on Windows?
1
vote
4answers
99 views
“Multipass” scripted modification of large file in-place (file-system level)?
I have just came to the problem of having to cut some lines from a large (gigabyte) sized file, and being aware of potential CPU hog trying to read it in memory, I wanted to edit it in-place ...
0
votes
3answers
3k views
How to copy files from linux to windows using winscp from a folder which contains millions of files
I need to copy files from a linux machine to a windows machine where the only ports which can be open are for SSH (22).
I can connect to the linux machine using WinSCP but the problem is once I try to ...
0
votes
0answers
76 views
How to solve 2038 swfupload error?
I use swfupload on my debian server. I have already increased limits in my php options too.It gives me 2038 error after 500-600MB uploading. I can upload files with 200mb, or 300MB, but can't upload ...
