27

I have a script that will pipe its output to |tee scriptnameYYMMDD.txt. After each cycle of the for loop in which the output is generated, I'll be reversing the file contents with tac scriptnameYYYYMMDD.txt > /var/www/html/logs/scriptname.txt so that the log output is visible in a browser window with the newest lines at the top.

I'll have several scripts doing this in parallel. I'm trying to minimize the disk activity, so output from |tee scriptnameYYYYMMDD.txt to a RAMdisk would be best. mktemp creates a file in the /tmp folder, but that doesn't appear to be off-disk.

  • It need to be a ramfs or a tmpfs (second is better). Check if a such fs is mounted on your system already, if yes you can use that. If no, you need to mount it. – peterh Mar 6 '15 at 6:16
  • I'd update fstab in order to mount a tmpfs on /tmp and then reboot the machine. – kasperd Mar 6 '15 at 7:51
  • 1
    you might consider incrementally tailing the files (or initiating it via CGI upon request or something) rather than tacing the whole thing. – mikeserv Mar 6 '15 at 8:37
  • This seems to work in testing (sorry for the diminished formatting): TEMPPATH="/ramdisk" LOGPATH="/var/www/html/log" ... echo <various calls to echo> | tee -a $TEMPPATH/moveKRT$(date '+%Y%m%d').txt ... at finish of for loop, I have cp $TEMPPATH/moveKRT$(date '+%Y%m%d').txt $LOGPATH/moveKRT$(date '+%Y%m%d').txt tac $TEMPPATH/moveKRT$(date '+%Y%m%d').txt > $LOGPATH/moveKRT.txt I'm aware of the miniscule possibility that the change from 23:59:59.999 to 00:00:00 may affect some files, but the chance is acceptable. – user208145 Mar 6 '15 at 8:54
  • Update on my progress on this: I have a 1 GB tmpfs volume mounted at /ramdisk. I have files coming into user home folders via FTP. They are moved to a subfolder /ramdisk/queues/xyz/ to process and are deleted afterwards. On startup, I have a script that re-creates my needed directory structure under /ramdisk. This should result in minimal disk activity for the incoming files. So far the only other way I see to reduce disk I/O for these scripts would be to create those user folders in a tmpfs created on startup, before any files come in via FTP to those user folders. Thanks all. – user208145 Mar 19 '15 at 20:36
22

You can mount a tmpfs partititon and write the file there:

mount -t tmpfs -o size=500m tmpfs /mountpoint

This partition now is limited to 500 MB. If your temporary file grows larger than 500 MB an error will occur: no space left on device. But, it doesn't matter when you specify a larger amount of space than your systems RAM has. tmpfs uses swap space too, so you cannot force a system crash, as opposed to ramfs.

You can now write your file into /mountpoint:

command | tee /mountpoint/scriptnameYYYYMMDD.txt
  • 3
    size=[num]% is also fine - a tmpfs can be limited to a percentage of available memory. It is usually easier that way if the commands are not always targeted for the same (real/virtual) machine. Also handy is that a tmpfs doesn't drop contents w/ -o remount,size=newsize. – mikeserv Mar 6 '15 at 8:34
  • 1
    Since the question revolves around a bash script, which we have no reason to assume is run by root, can you specify whether one could get around mount not allowing non-root users to use -t or -o? That is, how can one build scripts that use tmpfs (or RAM in any other fashion) to save on disk operations, if a suitable filesystem doesn't already exist on the system? – Jonathan Y. Jan 22 '16 at 11:59
  • Not a good answer. Will fail if you are not root. – Alex Jan 4 '17 at 0:27
11

Try this with Ubuntu:

ramtmp="$(mktemp -p /dev/shm/)"
tac scriptnameYYYYMMDD.txt > "$ramtmp"
  • 3
    how stable is writing to /run/shm/? Can I apply chmod changes to the temporary file if needed? I noticed that the folder's mode bits are identical to the /tmp folder. thanks. – user208145 Mar 6 '15 at 6:42
  • 1
    As for other directories, files generated in /run/shm (symlink to /dev/shm) by mktemp also have a default permission of -rw------- (600) and you can change it if needed with sudo chmod. – KrisWebDev Mar 6 '16 at 13:48
9

The following answer was discovered by investigating the previous answers and the info in this question here and would not have been found without them. Cudos to them.

On my linuxmint system (and I would assume most ubuntu based systems and possibly debian based too) there is a user owned tmpfs mounted automatically on /run/user/1000/

Use df -T to check.

11:41:11 jesse@Limbo:~$ df -T
Filesystem     Type      1K-blocks       Used Available Use% Mounted on
udev           devtmpfs   15904812          4  15904808   1% /dev
tmpfs          tmpfs       3184120       1700   3182420   1% /run
/dev/sdb2      ext4       14248880   11464788   2037240  85% /
none           tmpfs             4          0         4   0% /sys/fs/cgroup
none           tmpfs          5120          0      5120   0% /run/lock
none           tmpfs      15920584        848  15919736   1% /run/shm
none           tmpfs        102400         12    102388   1% /run/user
/dev/sdb3      ext4      100861352   90755700   4959136  95% /mnt/data

Under /run/user/ there is a directory for each normal user on the system

12:07:35 jesse@Limbo:~$ ls -l /run/user
total 0
drwx------ 2 root  root   40 Aug  7 09:50 0
drwx------ 8 jesse jesse 180 Aug  7 11:38 1000

These directories are named after their respective user's ids. We can get the user id with id -u see man id for details on this command.

12:07:43 jesse@Limbo:~$ ls -l /run/user/$(id -u)
total 0
drwx------ 2 jesse jesse  60 Aug  7 09:50 dconf
dr-x------ 2 jesse jesse   0 Aug  7 09:50 gvfs
drwx------ 2 jesse jesse  80 Aug  7 09:50 pulse
lrwxrwxrwx 1 root  root   17 Aug  7 09:50 X11-display -> /tmp/.X11-unix/X0

We can then use the mktemp command with the --tmpdir option to create temp files and directories in this tempfilesystem thus creating tempfiles in RAM.

Following the advice given here I create a temp directory first and then create my temp files in that:

mydir=$(mktemp -dt "$(basename $0).XXXXXXXX" --tmpdir=/run/user/$(id -u))

to create a temp directory /run/user/1000/bash.w42BYxbG/ then

myfile=$(mktemp -t "$(basename $0).XXXXXXXX" --tmpdir=$mydir)

to create a tempfile in it.

This makes cleaning up these file easy since all I have to do is rm -r $mydir.

By default all these files are owned and readable only by the user who created them.

Note: The $(basename $0) portion of the command extracts the name of the script/process that executed mktemp. If I have a script /home/jesse/scripts/myScript.sh then $(basename $0) returns myScript.sh when executed by this script. Thus the above commands would create /run/user/1000/myScript.sh.w42BYxbG/ and /run/user/1000/myScript.sh.w42BYxbG/myScript.sh.BCzSmq06 respectively.

  • /run/user/1000 is available on each linux which uses systemd, so it's not only available on debian-based systems, but as well on most other linux distributions. If not, /tmp should be used instead. You can simplify your commands by using the variable $XDG_RUNTIME_DIR which already keeps the correct directory for each user. For more info, check e.g. this answer: unix.stackexchange.com/questions/162900/… – Alex Jan 6 '17 at 14:51
  • 1
    Use mkstemp over mktemp !! Reason from: linux.die.net/man/3/mktemp : in the section Bugs " ... every use of mktemp() is a security risk, The race is avoided by mkstemp(3)." – Alex Jan 6 '17 at 15:07
0

As I understand it, the goals are: (1) do store the reversed file to disk, to be served to the browser (note: this could be tmpfs, etc, as all the other answers already detail); but, (2) avoid writing the first tmp file to disk, with the original output; and yet, (3) still display the original output to stdout.

If so, then the following might meet your needs, using bash process substitution (i.e., basically a named pipe):

command | tee >( tac > /var/www/html/logs/scriptname.txt )

Note that this is continuously printing output to stdout, and when the command ends (when your loop terminates), reverses that output and writes it to the output file. Illustrated via the following, which prints one line per second, and after termination the file /tmp/foo.txt contains the reversed lines:

i=0
while [ $i -lt 10 ]; do
    ((i = i+1))
    echo "==$i== $(date)"
    sleep 1
done | tee >( tac >> /tmp/foo.txt ) 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.