1

my Debian server disk maxed out because of a large postgresql log file and while I have deleted it, it is still held by postgresql. When I restart postgresql I get an error as the disk is full and the software cannot start. This is the file listed using lsof +L1:

COMMAND     PID     USER   FD   TYPE DEVICE    SIZE/OFF NLINK    NODE NAME
testproxy 22712 postgres    2w   REG    8,1 15309393920     0 1184540 /tmp/postgresql-9.4-main.log (deleted)

I've tried some commands suggested in other threads but it's not working. Can anyone suggest how to remove this file, bearing in mind restarting postgresql is not working?

thanks!

1
  • 1
    the logfile seems to be held open by command testproxy (running as user postgres) not by postgres itself. try killing testproxy
    – cas
    Dec 2 '15 at 0:51
6

You have a bigger problem than Out of Disk my friend!

This is a User Defined Function exploit which takes advantage of PostgreSQL's large objects. (lo_) functions.

On my server it is a trojan that creates a proxy to baby0119.com over port 80. It was installed as your postgres user over your postgres port 5432.

Check your 'postgres' database for a function called 'exec111'. \df+ exec111.

Drop that function, tighten up your pg_hba.conf, firewall, etc.

Also, check your postgresql log for commands issued or ERRORs.

Files I found on my box in /tmp are:

  • 6 Dec 7 11:37 sjkpppp
  • 961472 Dec 7 16:36 testproxy6
  • 8088 Dec 7 16:36 testproxy.so

If you have a web server running on the postgres server, check your web access logs, grep for 'proxytest' or proxy, etc.

2
  • Wow -- thanks for dropping by 1966Mustang! I indeed found both those things. Highly appreciate the warning!
    – Encomium
    Dec 9 '15 at 13:22
  • Hey guys, thanks for your help, we got the same problem. The question is still there how one can break our connection password ? Dec 14 '15 at 12:56
1

Or, if you're feeling particularly venturesome, gdb!

% lsof | grep deleted | grep deleteme
% perl -E 'while(1){ say "om nom nom"; sleep 1 }' > deleteme & rm deleteme
[2] 15720
% lsof | grep deleted | grep deleteme                                     
perl      15720    jdoe42    1w      REG                8,2        0    5376141 /home/jdoe42/deleteme (deleted)
% gdb -q -p 15720
...
(gdb) call close(1)
$1 = 0
(gdb) quit
...
% lsof | grep deleted | grep deleteme
% jobs
[1]  - running    perl -E 'while(1){ say "om nom nom"; sleep 1 }' > deleteme
% kill %1
% 
[1]  + terminated  perl -E 'while(1){ say "om nom nom"; sleep 1 }' > deleteme
% 

This however may or may not work, may break the program thus fiddled with in unexpected ways, cause hair loss, sudden use of Windows syndrome, et cetera. In other words, use at your own risk. Simply killing the program will most often be a vastly better option.

Key points are obtaining the file descriptor number (via the lsof or equivalent), which here is the STDOUT_FILENO (1w according to lsof) because that's what the shell redirected, and then to call close(2) on that file descriptor. Files opened by the program itself will likely have higher descriptor numbers (three and up); the question shows that standard error is going to the /tmp file (which appears to be a local security flaw, to write a static filename under /tmp like that).

0

For whatever reason, there is still a process (live or not) which has the file open. If it is not a zombie, you can kill it explicitly (with the right permissions):

sudo kill -9 22712
2
  • 3
    kill -9 is overkill.
    – cas
    Dec 2 '15 at 0:53
  • That's it -- thanks a lot for the suggestion!
    – Encomium
    Dec 2 '15 at 0:54
0

1966Mustang's answer is right.

  1. This is usually because your Postgres password is too weak.
  2. Check your pg_hba.conf, if you trusted all IP to connect your Server.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.