Set monthly data limit for internet proxy (squid) users

Question

We are using a squid proxy server (version 3.1.4) in our office environment on linux box. In total we are approximately 20 users and all are in one network and we have internet plan of 25GB/month. Now what we want is restrict each user to some size say 1GB/user/month (will be more better if it is more configurable, say 512MB for some users, 2GB for some users) , so that everyone get equal access.

I have searched a lot over internet like to restrict with bandwidth, or using maximum download size parameter reply_body_max_size, delay parameters. But cannot find what I want.

Is there any way to do this in squid or any other open source option?

Answer 1

I've been struggling to add this functionality as well. The problem with squish is, it's squishing users that is not supposed to be squished.

The workaround that I used was to use the daily data from lightsquid stored in /var/lib/lightsquid/report/<date>/.total. Since lightsquid data is updated every 10 minutes, it also gave my squid setup the ability of an almost "real-time" quota capability.

Answer 2

Probably via a custom helper that squid would call, say http://wiki.squid-cache.org/Features/Redirectors that looks at the client IP, consults a database of how much bandwidth they've used (perhaps parsed from the squid logs by something else), and redirects them to a sorry page if over. This will probably require a bunch of coding, if such code does not yet exist.

Answer 3

According to squid wiki page user data quota is under development (see below link).

squish can be used to limit data per IP per month. (using squid)

Features/Quota - Squid Web Proxy Wiki http://wiki.squid-cache.org/Features/Quota

Answer 4

After searching for more than a week, I found the solution in my way using html reports of sarg - Squid Analysis Report Generator, and extracting its data using simple bash script.

sarg stores its data in /var/www/sarg directory. To generate monthly report we have to manually use this command: sarg -d 01/08/2015-31/08/2015.

After this command I have written a simple script to block by ip-address which are overlimit as follows:

#!/bin/bash
range=2015Aug03-2015Aug25
ip=( 192.168.3.111 192.168.3.49 192.168.3.53 )
maxsize=100000000
cd /var/www/sarg/$range/

for system in "${ip[@]}"; do
        size=`cat $system/$system.html | grep TOTAL | awk '{print $4}' | tr -d 'class="header2"></t.<t'| sed 's/K/0/' | sed 's/M/0000/'`
        if [ "$size" -gt "$maxsize" ]
            then
                iptables -A INPUT -s $ip -p tcp --dport 3128 -j DROP
                iptables -A OUTPUT -s $ip -p tcp --dport 3128 -j DROP
                echo "$system blocked because data usage over $size bytes"
            else
                echo "$system used size is $size"
        fi
done
service iptables save

NOTE: This script is not fully furnished as of now and have to work little more for getting all things fully automated and flexible.

Answer 5

if you want to use SQUID only you can use NFQUEUE. you can redirect all traffic to NFQEUE first and control traffic in user space programming. Then you can decide DROP or or forward to SQUID (3128) port. You should create small prepaid charging system. see my small article for this.

But I suggest use RADIUS (dolaRadius) solution to achieve your requirement. There are possibility to integrate SQUID with FreeRADIUS and control traffic.

Answer 6

Squid stores the logs in the access.log file in this format by default:

time elapsed remotehost code/status bytes method URL rfc931 peerstatus/peerhost type

what we need here is the remotehost which is the IP address of the client, and bytes which is the bytes received (downloaded) during the request.

If you're wondering, Squid counts the bytes for the CONNECT requests too.

You can code a script in any programming language to find the lines containing the client's IP address, then for each line, parse it to get the bytes and add them together. After that you have to remove the access.log file so your script doesn't count the counted bytes again during the next script execution... Or instead of removing the access.log file you can make use of time (in the log format) which is the timestamp of the request, and let your script count the bytes after the last script execution's timestamp. Or you can use grep to find the lines containing the client IP and sed to remove them from the access.log file, see https://stackoverflow.com/a/11797762/8524395 and https://stackoverflow.com/a/5410784/8524395

You can even configure the log format to make it easier during coding. See http://www.squid-cache.org/Doc/config/logformat/ and https://wiki.squid-cache.org/Features/LogFormat

And you can also count the quota used by username if you don't need to do it by the client IP address. Do the same idea but first modify your logformat to contain the username, see:

        un  User name (any available)
        ul  User name from authentication
        ue  User name from external acl helper
        ui  User name from ident
        un  A user name. Expands to the first available name
            from the following list of information sources:
            - authenticated user name, like %ul
            - user name supplied by an external ACL, like %ue
            - SSL client name, like %us
            - ident user name, like %ui

Bytes test

I've downloaded an exactly 100MB file through the Squid server using curl. The access.log file logged the request as 105048163 bytes (105.048163 MB). The same request was made again and the same bytes amount was logged.

Also the same test was done for an exactly 1GB (1000MB) file. Squid logged it as 1050436965 bytes (1050.436965 MB)