0

Randomly our production site encounters a 503 error; each time after a varnish service restart, the requested site comes up normally.

I looked at the varnish log and all I see is "Service Unavailable" but nothing else.

I'm posting a portion of the varnishlog, the default.vcl and the varnish.params

  • << BeReq >> 857875
  • Begin bereq 857874 fetch
  • Timestamp Start: 1553896092.649343 0.000000 0.000000
  • BereqMethod GET
  • BereqURL /huf_en_us/search/ajax/suggest/?_=1553896063503&q=pot+leaf+sock
  • BereqProtocol HTTP/1.0
  • BereqHeader X-Real-IP: 172.16.204.112
  • BereqHeader X-Forwarded-Proto: https
  • BereqHeader SSL-OFFLOADED: https
  • BereqHeader X-Forwarded-Port: 443
  • BereqHeader Host: www.hufworldwide.com
  • BereqHeader CF-IPCountry: US
  • BereqHeader CF-RAY: 4bf50bf2cb163852-LAX
  • BereqHeader CF-Visitor: {"scheme":"https"}
  • BereqHeader accept: application/json, text/javascript, /; q=0.01
  • BereqHeader x-requested-with: XMLHttpRequest
  • BereqHeader accept-language: en-us
  • BereqHeader listrak-listening: 1
  • BereqHeader referer: https://www.hufworldwide.com/huf_en_us/triple-triangle-pullover-hoodie-pf00100?color=tropical-green
  • BereqHeader dnt: 1
  • BereqHeader user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1
  • BereqHeader x-newrelic-id: VQcDV1JVABAEUVJVDgkOXw==
  • BereqHeader cookie: store=huf_en_us; _vuid=5759a66b-8175-43cc-a614-51a32c0d7a08; PHPSESSID=los3vfdipo3ako096qoo830m05; form_key=QY53qr4aLFMypLqi; private_content_version=927393d221b9f95d0d23c147b7a38d6b; section_data_ids=%7B%22directory-data%22%3A1553894394%2C%22cus
  • BereqHeader CF-Connecting-IP: 2606:a000:1216:8689:a5a4:dde1:69bd:10ac
  • BereqHeader CF-Pseudo-IPv4: 252.241.164.167
  • BereqHeader True-Client-IP: 2606:a000:1216:8689:a5a4:dde1:69bd:10ac
  • BereqHeader CDN-Loop: cloudflare
  • BereqHeader X-Forwarded-For: 172.16.204.112, 127.0.0.1
  • BereqHeader Accept-Encoding: gzip
  • BereqProtocol HTTP/1.1
  • BereqHeader X-Varnish: 857875
  • VCL_call BACKEND_FETCH
  • VCL_return fetch
  • BackendOpen 54 web02(172.16.204.102,,80) 172.16.204.104 60836
  • Backend 54 d web02(172.16.204.102,,80)
  • Timestamp Bereq: 1553896092.649819 0.000476 0.000476
  • Timestamp Beresp: 1553896093.563128 0.913785 0.913309
  • BerespProtocol HTTP/1.1
  • BerespStatus 503
  • BerespReason Service Unavailable
  • BerespHeader Date: Fri, 29 Mar 2019 21:48:12 GMT
  • BerespHeader Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.0.30
  • BerespHeader X-Powered-By: PHP/7.0.30
  • BerespHeader X-Content-Type-Options: nosniff
  • BerespHeader X-XSS-Protection: 1; mode=block
  • BerespHeader X-Frame-Options: SAMEORIGIN
  • BerespHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT
  • BerespHeader Cache-Control: no-store, no-cache, must-revalidate
  • BerespHeader Pragma: no-cache
  • BerespHeader Set-Cookie: PHPSESSID=los3vfdipo3ako096qoo830m05; expires=Sun, 28-Apr-2019 21:48:13 GMT; Max-Age=2592000; path=/; domain=www.hufworldwide.com; secure; HttpOnly
  • BerespHeader X-NewRelic-App-Data: PxQGUFJVDAUIR1VXBAYOXl0IFB9AMQYAZBBZDEtZV0ZaClc9HiBQFg1ZWT1JJUpcXhAiDVlFRQkIXVNBPkkuA1cHVhZXZGh0QQRUBEYOQQk4anYRFj9kdUILDxZ0XlkSFl5aXwcUPz55DEwHSltRQkcKS0MdUR1SVAYHUUpTFgoAXVRQGxwGSkYCAwZbUltQB1oMWw0MBQVWRxUHUA1ABzk=
  • BerespHeader Connection: close
  • BerespHeader Transfer-Encoding: chunked
  • BerespHeader Content-Type: text/html; charset=UTF-8
  • TTL RFC -1 -1 -1 1553896094 1553896094 1553896092 375007920 0
  • VCL_call BACKEND_RESPONSE
  • TTL VCL 0 10 0 1553896094
  • VCL_return deliver
  • BerespHeader Content-Encoding: gzip
  • Storage malloc Transient
  • ObjProtocol HTTP/1.1
  • ObjStatus 503
  • ObjReason Service Unavailable
  • ObjHeader Date: Fri, 29 Mar 2019 21:48:12 GMT
  • ObjHeader Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.0.30
  • ObjHeader X-Powered-By: PHP/7.0.30
  • ObjHeader X-Content-Type-Options: nosniff
  • ObjHeader X-XSS-Protection: 1; mode=block
  • ObjHeader X-Frame-Options: SAMEORIGIN
  • ObjHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT
  • ObjHeader Cache-Control: no-store, no-cache, must-revalidate
  • ObjHeader Pragma: no-cache
  • ObjHeader Set-Cookie: PHPSESSID=los3vfdipo3ako096qoo830m05; expires=Sun, 28-Apr-2019 21:48:13 GMT; Max-Age=2592000; path=/; domain=www.hufworldwide.com; secure; HttpOnly
  • ObjHeader X-NewRelic-App-Data: PxQGUFJVDAUIR1VXBAYOXl0IFB9AMQYAZBBZDEtZV0ZaClc9HiBQFg1ZWT1JJUpcXhAiDVlFRQkIXVNBPkkuA1cHVhZXZGh0QQRUBEYOQQk4anYRFj9kdUILDxZ0XlkSFl5aXwcUPz55DEwHSltRQkcKS0MdUR1SVAYHUUpTFgoAXVRQGxwGSkYCAwZbUltQB1oMWw0MBQVWRxUHUA1ABzk=
  • ObjHeader Content-Type: text/html; charset=UTF-8
  • ObjHeader Content-Encoding: gzip
  • Fetch_Body 2 chunked -
  • Gzip G F E 16777 6581 80 52568 52578
  • Timestamp BerespBody: 1553896093.572716 0.923373 0.009588
  • BackendClose 54 web02(172.16.204.102,,80)
  • Length 6581
  • BereqAcct 3058 0 3058 869 16789 17658
  • End

default.vcl

vcl 4.0;

import std; import directors;

REM The minimal Varnish version is 4.0 For SSL offloading, pass the following header in your proxy server or load balancer: 'SSL-OFFLOADED: https'

backend web01 { .host = "172.16.204.101"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

backend web02 { .host = "172.16.204.102"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

backend web03 { .host = "172.16.204.120"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

backend web04 { .host = "172.16.204.121"; .port = "80"; .connect_timeout = 1600s; .first_byte_timeout = 1600s; .between_bytes_timeout = 1600s; .max_connections = 20000; }

varnish.params

REM Varnish environment configuration description. This was derived from REM the old style sysconfig/defaults settings

REM Set this to 1 to make systemd reload try to switch VCL without restart. RELOAD_VCL=1

REM Set WARMUP_TIME to force a delay in reload-vcl between vcl.load and vcl.use REM This is useful when backend probe definitions need some time before declaring REM configured backends healthy, to avoid routing traffic to a non-healthy backend. REM WARMUP_TIME=0

REM Main configuration file. You probably want to change it. VARNISH_VCL_CONF=/etc/varnish/default.vcl

REM Default address and port to bind to. Blank address means all IPv4 REM and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted REM quad, or an IPv6 address in brackets. REM VARNISH_LISTEN_ADDRESS=192.168.1.5 VARNISH_LISTEN_PORT=8080

REM Admin interface listen address and port VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082

REM Shared secret file for admin interface VARNISH_SECRET_FILE=/etc/varnish/secret

REM Backend storage specification, see Storage Types in the varnishd(5) REM man page for details. VARNISH_STORAGE="malloc,24G"

REM User and group for the varnishd worker processes VARNISH_USER=builder VARNISH_GROUP=builder

REM Other options, see the man page varnishd(1) DAEMON_OPTS="-p thread_pool_min=200 -p thread_pool_max=2000 -p thread_pool_timeout=600 -p http_resp_hdr_len=90000 -p http_max_hdr=200"

  • 503 responses typically indicate the server is overloaded. This issue generally is a result of heavy load rather than server issues and goes away on it's own so long as the load reduces. – Jesse_b Mar 30 at 13:13
  • which server? varnish server or web nodes? – user179185 Apr 1 at 16:01
  • I don't even know what a varnish server is but http responses are fairly standardized across all applications. – Jesse_b Apr 1 at 17:07
  • the reason I'm asking this is because the architecture is as follow: Load Balancer -> Varnish Server + Nginx Reverse Proxy -> Web nodes (4) with Apache So when you say "the server is overloaded", does it mean the web nodes are overloaded or the vanish server itself is overloaded – user179185 Apr 1 at 20:39
  • Monitor the performance on all your servers and find out which one(s) is/are experiencing high load at the time of a 503 – Jesse_b Apr 1 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.