The tag has no wiki summary.

learn more… | top users | synonyms

3
votes
1answer
55 views

GNU Parallel - grepping n lines for m regular expressions

The GNU parallel grepping n lines for m regular expressions example states the following: If the CPU is the limiting factor parallelization should be done on the regexps: cat regexp.txt | ...
3
votes
3answers
206 views

How can I get the first column of a 300GB txt file?

let me first explain my problem in detail. It's actually pretty simple. I have a huge .txt file, 300GB to be more preciese, and I would like to put all the distinct strings from the first column, that ...
1
vote
1answer
180 views

Best parameters for ext4 filesystem to handle virtualbox disk images

I have ext4 partition, which holds disk image files VirtualBox works with. They all are fixed-size images (i.e. files never change their size). They are defragmented as much as possible (with ...
3
votes
1answer
70 views

Is it possible to get file list, byte range from the head of a tar+bzip2 file?

Essentially, I'm hoping someone with an advanced knowledge of tar/bz2 can answer whether this is possible. The situation is that we have a periodic 24GB feed of data from a vendor, as a .tbz file. ...
2
votes
3answers
175 views

Filesystem surprisingly large

I'm running a dual booted Windows 7 with Linux Mint right now and I just checked my filesystem size and it jumped from quite small (about 30 GB) to 200 GB recently (no clue what triggered this, I just ...
3
votes
2answers
898 views

Basic sed command on large one-line file: couldn't re-allocate memory

I have a 250 MB text file, all in one line. In this file I want to replace a characters with b characters: sed -e "s/a/b/g" < one-line-250-mb.txt It fails with: sed: couldn't re-allocate ...
5
votes
1answer
102 views

Merge nonzero blocks of huge (sparse) file A into huge file B

I have two partial disk images from a failing hard drive. File B contains the bulk of the disk's contents, with gaps where sector reads failed. File A is the result of telling ddrescue to retry all ...
3
votes
3answers
292 views

awk read error bad address

I have a file that is roughly 30GB. I am using, awk ' $1 <= 2000 ' myfile.txt >> myfile_new.txt But, I get this error after a while - awk: read error (Bad address) I guess this is ...
3
votes
1answer
500 views

Viewing large image on Linux

I need to view a large (50000x40000 px) png image on Linux. Unfortunately most tools (eog, convert etc.) either crashes or fail with note about too little memory. Is there a way to view this image (I ...
4
votes
4answers
166 views

how to sort access log efficiently in blocks

Access logs are more or less sorted by time, but to aggregate connections by time (uniq -c), you need to sort them a bit more. For a huge access log sort is very inefficient, because it buffers and ...
4
votes
2answers
474 views

Out of memory while using sed with multiline expressions on giant file

I am currently trying to remove all newlines that are not preceded by a closing parenthesis, so I came up with this expression: sed -r -i -e ":a;N;$!ba;s/([^\)])\n/\1/g;d" reallyBigFile.log It does ...
2
votes
2answers
13k views

Easy way to copy lines from one file to another [duplicate]

Possible Duplicate: cat line X to line Y on a huge file Very simple issue but can't seem to find a simple resolution! I have a massive text file from which I only need around 150 lines. ...
5
votes
7answers
1k views

Seeking alternative GUI File editor with large file support

Can anyone recommend a file editor, preferably free, that handles large files as well as EmEditor on Windows? I'm on Ubuntu, but other people reading this might not be, so suggestions for other Unix ...
6
votes
3answers
415 views

Emacs: Open a buffer with all lines between lines X to Y from a huge file

In the same spirit as this other question: cat line X to line Y on a huge file: Is there a way to open from within Emacs (and show on a buffer) a given set of lines (e.g. all lines between line X ...
28
votes
5answers
38k views

cat line X to line Y on a huge file

Say I have a huge text file (>2GB) and I just want to cat the lines X to Y (e.g. 57890000 to 57890010). From what I understand I can do this by piping head into tail or viceversa, i.e. head -A ...
8
votes
2answers
5k views

largefile feature at creating file-system

Is useful to use -T largefile flag at creating a file-system for a partition with big files like video, and audio in flac format? I tested the same partition with that flag and without it, and using ...
3
votes
1answer
101 views

ed: set first line as the default current line

I am trying to use ed to edit the first line of a large file, but I have to wait for ed to read all lines. Is there any way I could stop ed from reading the whole file and start editing immediately, ...
1
vote
4answers
131 views

“Multipass” scripted modification of large file in-place (file-system level)?

I have just came to the problem of having to cut some lines from a large (gigabyte) sized file, and being aware of potential CPU hog trying to read it in memory, I wanted to edit it in-place ...
3
votes
2answers
1k views

how to find offset of one binary file inside another?

I have two binary files. One of few hundreds kilos and other of few gigabytes. I want to know whether the whole, smaller, file is contained within the larger one and if so then what is the offset from ...
1
vote
4answers
1k views

Grepping over a huge file performance

I have FILE_A which has over 300K lines and FILE_B which has over 30M lines. I created a bash script that greps each line in FILE_A over in FILE_B and writes the result of the grep to a new file. ...
37
votes
5answers
21k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
4
votes
1answer
697 views

Behavior of mmap'd memory on memory pressure

I have a large tar file (60GB) containing image files. I'm using mmap() on this entire file to read in these images, which are accessed randomly. I'm using mmap() for the following reasons: Thread ...
0
votes
3answers
5k views

How to copy files from linux to windows using winscp from a folder which contains millions of files

I need to copy files from a linux machine to a windows machine where the only ports which can be open are for SSH (22). I can connect to the linux machine using WinSCP but the problem is once I try to ...
2
votes
3answers
351 views

Uncopyable 5GB file, BTRFS

I have a 5.3GB tar file, located on a BTRFS partitioned eSATA drive. When trying to copy the file to another disk, or expand the tar the process seems to freeze. No errors are generated, and waiting ...
2
votes
1answer
3k views

Edit really big files with vi

How can I edit a really big file with vi? e.g.: a 20 GByte log file. Are there any modifications for it to handle this size?
26
votes
7answers
5k views

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
3
votes
1answer
607 views

Can I safely delete ~/Trash file?

My Uni account is over disk quota and requesting more quota takes time. Unfortunately, I need disk space now and I noticed that the Trash file in my home directory is quite a large file. Now I’m a ...
5
votes
1answer
285 views

What linux editor can open a 200G file for editing within a minute or two?

I'm looking for an editor that will open the file in chunks (not try to read the whole file into memory) as I'm trying to hand-edit a 200G file.
13
votes
1answer
6k views

Number of files per directory

I have a directory with about 100000 small files (each file is from 1-3 lines, each file is a text file). In size the directory isn't very big (< 2GB). This data lives in a professionally ...
6
votes
2answers
6k views

Diffing two big text files

I have two big files (6GB each). They are unsorted, with linefeeds (\n) as separators. How can I diff them? It should take under 24h.
15
votes
5answers
3k views

How can I edit a large file in place?

I have a few files sized > 1 GB each. I need to remove last few bytes from the files. How can I do it? I prefer to edit file in place to save disk space. I am on HP-UX.