Tag Info

Hot answers tagged

14

The problem, of course, is that you run grep on the big file 10,000 times. You should read both files only once. If you want to stay outside scripting languages, you can do it this way: Extract all numbers from file 1 and sort them Extract all numbers from file 2 and sort them Run comm on the sorted lists to get what's only on the second list Something ...


10

SQLite's small size and levels of completeness, stability & speed make it a popular choice for low-resource environments, which embedded systems usually are. It is used by parts of the current iPhone, Android and Symbian phone operating systems for this reason. You might want to add some details to your question to get more specific answers: do you know ...


8

This answer is based on the awk answer posted by potong.. It is twice as fast as the comm method (on my system), for the same 6 million lines in main-file and 10 thousand keys... (now updated to use FNR,NR) Although awk is faster than your current system, and will give you and your computer(s) some breathing space, be aware that when data processing ...


8

I don't think there is a standard tool for that. Except for grep/awk/sed etc. But using this you will need to care about lot of other issues like locking, format, special characters, etc. I suggest to use sqlite. Define a simple table and then create tool_get() and tool_put() shell functions. sqlite is portable, fast. You will get extra flexibility for ...


8

BDB (libdb) has historically been the embedded database of choice for many applications, shipping with most UNIXes and used by lots of software. If you're accustomed to SQL relational databases, though, BDB is not one - it is simply a (really good) key-value store. SQLite is a different popular embedded database. As the name implies, it is a SQL database ...


7

Yes, definitely do use a database. They're made exactly for tasks like this.


6

Since you named it, the standard redis client has a command line interface through redis-cli. Some examples from redis-cli -h: cat /etc/passwd | redis-cli -x set mypasswd redis-cli get mypasswd redis-cli -r 100 lpush mylist x (And if you want to access the db through the filesystem you can use sockets with -s. A tool that would read the db index ...


5

You're right the documentation about unixODBC still rare. For the configurations files, unixODBC use only two config files: /etc/odbcinst.ini : Here you define the driver /etc/odbc.ini : Informations about connections You can find a great documentation about installing this drivers and libraries on various linux systems here: ...


5

On Ubuntu you should use service: sudo service mysql start And in the future if and when you want to restart it: sudo service mysql restart


5

When you do an LVM snapshot, pending data (in kernel buffers, not applications') is flushed to disk and applications are blocked from writing while the snapshotting is ongoing. You can also freeze a FS in that same way if you're backuping the block device the FS is on by some external means (like the disk is virtual and you're backing it up on the host) ...


4

dbmutil might get you what you want. It has shell utilities for the operations you describe in the question. I wouldn't say it's exactly standard, but it does have the facilities you want.


4

Don't rename the databases directories directly. Instead, dump the database out, create a new database name (which, in turn, will create the new directory name) and then dump the database back into the new database name: mysqldump -u username -p olddatabase > olddbdump.sql mysqladmin -u username -p create newdatabase mysql -u username -p newdatabase ...


4

If your database is small enough, then you can use the filesystem. The advantage of this approach is that it's very low-tech, and will work everywhere with very little code. If the keys are composed of printable characters and do not contain /, then you can use them as file names: put () { key=$1; value=$2; printf %s "$value" >"datastore.db/$key"; } ...


4

MySQL stores DB files in /var/lib/mysql by default, but you can override this with configuration, typically stored in /etc/my.cnf, although Debian uses /etc/mysql/my.cnf.


3

In your example, backupdb is called a variable. A shell variable will be there until you change it to a new value, or the shell exits (most likely because you type exit, or close the terminal). In your case, backupdb will be there for a long time, so you don't need to type that again and again. My guess is that you are using the BASH shell because it's ...


3

This might work for you: awk '/^[0-9]/{a[$0]++}END{for(x in a)if(a[x]==1)print x}' file{1,2} >file3 EDIT: Amended script to allow for duplicates and unknown keys in both files, still produces keys from the first file not present in the second: awk '/^[0-9]/{if(FNR==NR){a[$0]=1;next};if($0 in a){a[$0]=2}}END{for(x in a)if(a[x]==1)print x}' file{1,2} ...


3

You have a cmd-line tool which does exactly this: mysqlhotcopy It works fine wy myisam tables, but not with InnoDb tables. If you have configured your server with lvm, and put your /var/lib/mysql on a dedicated volume here is the way I recommend to backup very fast and in a non-blocking way all your databases: mysql -U root -p > flush tables with ...


3

A quick test program shows errno = 17 is "File exists" (just an observation, not that I'd expect you to figure it out before posting). I found this thread on the mysql site, which suggests that you have files in /var/lib/mysql/my_db/ that mysql didn't create (or that it thinks it didn't create). List the files (you'll need to sudo) and see what's there. ...


3

How do I connect to my Linux-based servers? SSH is the de facto standard way of managing Linux-based servers. Is there something similar to Remote Desktop? Yes, NX (freeNX, or Nomachine NX) works over SSH, it's very common in enterprise environments. Also you can use VNC, or Citrix, and RDP is also possible. Do I need to use straight command line Linux ...


3

Putting each tuple on a separate line: sed 's@)\s*,\s*(@)\n(@g' your_file To apply modifications to the file (instead of printing the modified file to stdout): sed -i 's@)\s*,\s*(@)\n(@g' your_file To report only value3 (assuming a modified file): awk -F',' '{print $3}' <your_new_file This assumes the values themselves don't contain ,. See ...


3

The protocol is plain TCP/IP. From posgresql documentation about "frontends" and "backends" protocol: PostgreSQL uses a message-based protocol for communication between frontends and backends (clients and servers). The protocol is supported over TCP/IP and also over Unix-domain sockets. Port number 5432 has been registered with IANA as the customary TCP ...


3

There are numerous Open Source and free database options. Popular choices are: MySQL and it's successor MariaDB PostgreSQL ... You can install them from the commandline with: sudo apt-get install <mysql-server | mariadb-server | postgresql> or using the GUI package manager aptitude


2

$ ls -ld /var/lib/mysql drwx------ 21 mysql mysql 4096 2011-11-18 14:07 /var/lib/mysql Yes, you have no permissions for that directory (it may be root:root depending on your setup). Use: sudo mysql -u mysqluser -e "LOAD DATA INFILE '/var/lib/mysql/tmp/test.dat' INTO TABLE myapp.cars;" Or just copy the file to your home directory (or wherever) and chown ...


2

You'll find what you need directly in the mysqldump documentation: mysqldump -u username -p db_test table1 table2 ... > db_test.sql Load the dump the same way as if it was a full database dump.


2

Maybe Coppermine Photo Gallery is useful or can serve as a starting point: Coppermine is a multi-purpose fully-featured and integrated web picture gallery script written in PHP using GD or ImageMagick as image library with a MySQL backend. If you want to do it yourself, shouldn't be very hard to do with PHP or a web framework using Python, Ruby etc... ...


2

With that much data, you should really switch to a database. In the meantime, one thing you must do to get anywhere near decent performance is not to search file1 separately for each key. Run a single grep to extract all the non-excluded keys at once. Since that grep also returns lines that don't contain a key, filter those away. grep -o '[0-9]\{12\}' file2 ...


2

Permit me to reinforce what others have said, "Get thee to a database!" There are MySQL binaries freely available for most platforms. Why not SQLite? It's memory-based, loading a flat-file when you start it, then closing it when you're done. This means that if your computer crashes or the SQLite process goes away, so does all the data. Your problem looks ...


2

If you have installed the Linux server yourself or someone built it for you, I assume you have console access to the server. Start with finding out look for root password on that server. If you have built it yourself, you would know it. Otherwise you would have to get it from the person who built it for you. You need to go to the console of the server and ...


2

From what I understand and this is my assumption, you did not install or you are not "owner" of these servers. As such I strongly advise you to contact your system administrator as ask him how to connect to the systems. In any case, the most common way to connect to a Linux server is by ssh, this is CLI mode but you if it is properly configured you can ...


2

you can do this using sed #sed -ne 's/^\(.*\)\.$/\1/p' <data_in_file >data_out_file BTW, good manual using sed is sed one liners explained



Only top voted, non community-wiki answers of a minimum length are eligible