Tag Info

Hot answers tagged

14

The problem, of course, is that you run grep on the big file 10,000 times. You should read both files only once. If you want to stay outside scripting languages, you can do it this way: Extract all numbers from file 1 and sort them Extract all numbers from file 2 and sort them Run comm on the sorted lists to get what's only on the second list Something ...


10

SQLite's small size and levels of completeness, stability & speed make it a popular choice for low-resource environments, which embedded systems usually are. It is used by parts of the current iPhone, Android and Symbian phone operating systems for this reason. You might want to add some details to your question to get more specific answers: do you know ...


8

This answer is based on the awk answer posted by potong.. It is twice as fast as the comm method (on my system), for the same 6 million lines in main-file and 10 thousand keys... (now updated to use FNR,NR) Although awk is faster than your current system, and will give you and your computer(s) some breathing space, be aware that when data processing ...


8

I don't think there is a standard tool for that. Except for grep/awk/sed etc. But using this you will need to care about lot of other issues like locking, format, special characters, etc. I suggest to use sqlite. Define a simple table and then create tool_get() and tool_put() shell functions. sqlite is portable, fast. You will get extra flexibility for ...


8

BDB (libdb) has historically been the embedded database of choice for many applications, shipping with most UNIXes and used by lots of software. If you're accustomed to SQL relational databases, though, BDB is not one - it is simply a (really good) key-value store. SQLite is a different popular embedded database. As the name implies, it is a SQL database ...


6

Since you named it, the standard redis client has a command line interface through redis-cli. Some examples from redis-cli -h: cat /etc/passwd | redis-cli -x set mypasswd redis-cli get mypasswd redis-cli -r 100 lpush mylist x (And if you want to access the db through the filesystem you can use sockets with -s. A tool that would read the db index ...


4

Don't rename the databases directories directly. Instead, dump the database out, create a new database name (which, in turn, will create the new directory name) and then dump the database back into the new database name: mysqldump -u username -p olddatabase > olddbdump.sql mysqladmin -u username -p create newdatabase mysql -u username -p newdatabase ...


4

If your database is small enough, then you can use the filesystem. The advantage of this approach is that it's very low-tech, and will work everywhere with very little code. If the keys are composed of printable characters and do not contain /, then you can use them as file names: put () { key=$1; value=$2; printf %s "$value" >"datastore.db/$key"; } ...


4

When you do an LVM snapshot, pending data (in kernel buffers, not applications') is flushed to disk and applications are blocked from writing while the snapshotting is ongoing. You can also freeze a FS in that same way if you're backuping the block device the FS is on by some external means (like the disk is virtual and you're backing it up on the host) ...


4

You're right the documentation about unixODBC still rare. For the configurations files, unixODBC use only two config files: /etc/odbcinst.ini : Here you define the driver /etc/odbc.ini : Informations about connections You can find a great documentation about installing this drivers and libraries on various linux systems here: ...


3

In your example, backupdb is called a variable. A shell variable will be there until you change it to a new value, or the shell exits (most likely because you type exit, or close the terminal). In your case, backupdb will be there for a long time, so you don't need to type that again and again. My guess is that you are using the BASH shell because it's ...


3

You have a cmd-line tool which does exactly this: mysqlhotcopy It works fine wy myisam tables, but not with InnoDb tables. If you have configured your server with lvm, and put your /var/lib/mysql on a dedicated volume here is the way I recommend to backup very fast and in a non-blocking way all your databases: mysql -U root -p > flush tables with ...


3

This might work for you: awk '/^[0-9]/{a[$0]++}END{for(x in a)if(a[x]==1)print x}' file{1,2} >file3 EDIT: Amended script to allow for duplicates and unknown keys in both files, still produces keys from the first file not present in the second: awk '/^[0-9]/{if(FNR==NR){a[$0]=1;next};if($0 in a){a[$0]=2}}END{for(x in a)if(a[x]==1)print x}' file{1,2} ...


3

A quick test program shows errno = 17 is "File exists" (just an observation, not that I'd expect you to figure it out before posting). I found this thread on the mysql site, which suggests that you have files in /var/lib/mysql/my_db/ that mysql didn't create (or that it thinks it didn't create). List the files (you'll need to sudo) and see what's there. ...


3

How do I connect to my Linux-based servers? SSH is the de facto standard way of managing Linux-based servers. Is there something similar to Remote Desktop? Yes, NX (freeNX, or Nomachine NX) works over SSH, it's very common in enterprise environments. Also you can use VNC, or Citrix, and RDP is also possible. Do I need to use straight command line Linux ...


3

Putting each tuple on a separate line: sed 's@)\s*,\s*(@)\n(@g' your_file To apply modifications to the file (instead of printing the modified file to stdout): sed -i 's@)\s*,\s*(@)\n(@g' your_file To report only value3 (assuming a modified file): awk -F',' '{print $3}' <your_new_file This assumes the values themselves don't contain ,. See ...


3

The protocol is plain TCP/IP. From posgresql documentation about "frontends" and "backends" protocol: PostgreSQL uses a message-based protocol for communication between frontends and backends (clients and servers). The protocol is supported over TCP/IP and also over Unix-domain sockets. Port number 5432 has been registered with IANA as the customary TCP ...


2

$ ls -ld /var/lib/mysql drwx------ 21 mysql mysql 4096 2011-11-18 14:07 /var/lib/mysql Yes, you have no permissions for that directory (it may be root:root depending on your setup). Use: sudo mysql -u mysqluser -e "LOAD DATA INFILE '/var/lib/mysql/tmp/test.dat' INTO TABLE myapp.cars;" Or just copy the file to your home directory (or wherever) and chown ...


2

You'll find what you need directly in the mysqldump documentation: mysqldump -u username -p db_test table1 table2 ... > db_test.sql Load the dump the same way as if it was a full database dump.


2

With that much data, you should really switch to a database. In the meantime, one thing you must do to get anywhere near decent performance is not to search file1 separately for each key. Run a single grep to extract all the non-excluded keys at once. Since that grep also returns lines that don't contain a key, filter those away. grep -o '[0-9]\{12\}' file2 ...


2

Permit me to reinforce what others have said, "Get thee to a database!" There are MySQL binaries freely available for most platforms. Why not SQLite? It's memory-based, loading a flat-file when you start it, then closing it when you're done. This means that if your computer crashes or the SQLite process goes away, so does all the data. Your problem looks ...


2

Maybe Coppermine Photo Gallery is useful or can serve as a starting point: Coppermine is a multi-purpose fully-featured and integrated web picture gallery script written in PHP using GD or ImageMagick as image library with a MySQL backend. If you want to do it yourself, shouldn't be very hard to do with PHP or a web framework using Python, Ruby etc... ...


2

If you have installed the Linux server yourself or someone built it for you, I assume you have console access to the server. Start with finding out look for root password on that server. If you have built it yourself, you would know it. Otherwise you would have to get it from the person who built it for you. You need to go to the console of the server and ...


2

From what I understand and this is my assumption, you did not install or you are not "owner" of these servers. As such I strongly advise you to contact your system administrator as ask him how to connect to the systems. In any case, the most common way to connect to a Linux server is by ssh, this is CLI mode but you if it is properly configured you can ...


2

With GNU date (default on Linux Mint) you can do: mysqldump -uroot -p MyDatabase >/home/users/backup_MyDB/$(date +%F)_full_myDB.sql To delete files older than 1 week: find /home/users/backup_MyDB -type f -mtime +7 -exec rm {} + Although generally it is wise to see what you are deleting before you delete (at least when testing you script) for this ...


1

I finally solved the problem, running a query using another tool (not through MS Access or MS Excel) worked massively faster, ended up using DaFT (Database Fishing Tool) to SELECT INTO a text file. Processed all 50 million rows in a few hours. It seems the dll driver I was using doesn't work well with any MS products.


1

Some things to try, in no particular order: file(1) might be able to identify the files Any strange, not mounted disk partitions could be raw devices Rummage around in /etc/init.d (or whatever system it uses to start services), the database was probably started from there somehow. strings(1) applied to executables often help to identify callouts, version ...


1

With your rhel tag, I guess that you are using a Red Hat system. This error message tells you that you need a file named libodbcinst.a or libodbcinst.so. FreeTDS needs to link to it in order to work properly. On Red Hat System, pbone says that you need to install unixODBC-devel via your package system, if you want to pursue your manual installation. But ...


1

I agree with @jan-steinman that you should use a database for this kind of task. There are lots of ways to hack together a solution with a shell script as the other answers show, but doing it that way will lead to a lot of misery if you're going to use and maintain the code for any length of time more than just a one-day throw-away project. Assuming you're ...


1

Wasn't going to post my answer because I thought that such amount of data shouldn't be processed with a shell script, and the right answer to use a database was already given. But since now there are 7 other approaches... Reads the first file in memory, then greps the second file for numbers and checks if values are stored in memory. It should be faster ...



Only top voted, non community-wiki answers of a minimum length are eligible