Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Does anyone here know the resources it takes for editors to search and replace in a large file? The reason I am asking, is I have a 32 core server that HTOP is only showing one core at 100% when search/replace on a 3GB file with an editor. I am wondering if my editor search/replace is a single threaded, and if so is there a way to delegate more resources so that these tasks don't take so long? I have to say, it is frustrating to see 31 idle cores and 1 at 100% when a task is taking 25 to 30 minutes.

Oh and if it makes a difference, the RAM is 32GB sitting at 19GB used including cache.

share|improve this question
1  
Humph... why don't you split the file into 32 portions and run 32 instances of the script and then splice the results? (Which is, BTW, the whole point of Map/Reduce) –  Deer Hunter Feb 13 '13 at 18:44
2  
If you are doing a search and replace on a 3GB file, then I strongly suggest you look at using the stream editor. –  Zoredache Feb 13 '13 at 19:00
    
A few more commands for you to look for: in bash - wait, split -b, cat. –  Deer Hunter Feb 13 '13 at 19:15
add comment

migrated from serverfault.com Feb 13 '13 at 21:32

This question came from our site for professional system and network administrators.

4 Answers

An editor may or may not be multi-threaded, but, even if it is, it is unlikely to use threads for this purpose for one simple reason: doing so would not provide any advantages for normative use, and it would no doubt create developer headaches and possibly compromise features that are considered important (for normative use).

Given an infinite amount of time and an infinite number of programmers, no doubt all software would be wildly optimized down to the smallest most irrelevant details, extensively tested to ensure these optimizations do not negatively impact anything, etc. Sans that, no one wants to spend their time coding features 99.9% of users will never to appreciate, especially if the 0.1% who do do so because, eg, (an analogy) they really wanted to open soup cans with a hammer.

As a few people have pointed out, loading a 3 GB file into a text editor to do a search and replace is well, something you would do if the only way you knew how to do a search and replace is in a text editor. I'm not trying to insult you with that, BTW, just give you a friendly nudge -- now is the time to broaden some horizons ;)

share|improve this answer
add comment

Most likely, the editors are single threaded. You'd probably be better off splitting your file into 32 pieces and using something like perl or sed to search and replace.

share|improve this answer
add comment

Check out sed, the stream editor. It has a command set somewhat like vi, but doesn't read the file in to process it, it reads, modifies and writes out one line at a time (mostly, look at the manual). So you can at least reduce the time needed to read in the file (the editor has to build complex in-memory data structures) and then write it out.

[I'm (not so) surprised that the current batch of editors are able to process such files, I fondly remember that the original vi crashed badly with files that were a few dozen of KiB in size... sic transit gloria mundii.]

share|improve this answer
add comment

Search and replace over 3GB of text is a hard task for any editor. The best solution in my opinion, is to use Perl. You can use perl to automatically divide the file into smaller portions and run the regex in parallel on each portion. There is lots of way you can code this into perl. I will post one example later.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.