Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a table URL_Experiment in my database (mySQL database). I have 2 million URL links in this table.

For each URL, I am checking if some particular text is present in the URL and updating the information back in my URL_Experiment table.

Now, I need to make the script run fast. The easy solution that comes to my mind is,

Split the URL_Experiment into 10 tables (each with 200,000 rows) and run the script 10 times simultaneously. By this way, I can ensure that the updates happen properly. But for this, I need to save 10 scripts each accessing the correct table (I need to rename the tables from 1 to 10).

Though, the above seems a pretty neat solution, I am looking for more sophisticated solution. Something, which will execute the same script simultaneously (in parallel) as 10 processes. My concern is the execution time. I do not want the resources to be wasted when they can be utilized.

share|improve this question
    
No sir. If it is a spamming system, I might not have revealed my original identity. It is related to my thesis research work. I had built the entire code using excel VBA. But, it was failing for larger datasets and so I am rewriting the entire code in bash script. If you see my SO questions, most of them would be related to data extraction in VBA only. –  Ramesh Feb 13 at 0:20
1  
I am glad that am getting noticed for my questions :) I am really learning a lot these few days after starting to rewrite the code in script. I will mention all these links in my thesis defense :) –  Ramesh Feb 13 at 0:25
    
Are you populating the MySQL DB or is it the source of the URLs that you're being provided? I ask b/c I might be inclined to use sqlite instead of MySQL. BTW, what is your overall objective for your project? –  slm Feb 13 at 0:36
    
I generate the URLs based on the names that I get from another source. I know the website follows a particular pattern always and so I check if these URLs exist and if they exist, I extract the information from those URLs and store them in my MySQL database tables. –  Ramesh Feb 13 at 0:40
    
How was the performance in the creation of the MySQL DB? Often with this volume of rows the design of the architecture will bite you if you don't pay special attn. to it initially: stackoverflow.com/questions/1711631/… –  slm Feb 13 at 0:42

1 Answer 1

up vote 6 down vote accepted

You can try GNU parallel. Many examples can be found here and here.

I am a member of the Swift team and we do exactly these kinds of things for large research projects.

share|improve this answer
    
Thanks a lot for the suggestion. If I need to use GNU parallel, do I have to change the script or something? Do I need to make any configuration changes? –  Ramesh Feb 13 at 0:18
    
Ketan offered to contact him directly if you need help setting up Swift, moving this to the comments, not really part of the answer: "Feel free to contact me if you would like to setup Swift for your problem.". –  slm Feb 13 at 0:19
3  
parallel does seem like the best for the task, specifically I'd do something like mysql -e 'select ...' | parallel --pipe -j10 --round-robin ./scriptname.sh –  Patrick Feb 13 at 5:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.