It’s likely dogfrog4 has been undergoing a “slow http” attack (sometimes known as the “slowloris attack” after one of the tools that does it).

We’ve spent quite a bit of time overnight recompiling the web server to add mitigation for this; the mitigation involves timing out requests and banning IPs that are generating slow attacks.

The symptoms we were experiencing yesterday were consistent with a slow http attack – webserver stops responding, load average low, occasional response from webserver, restart fixes the problem, etc.

We’re keeping a careful eye on the situation.

Last night an upgrade to cPanel on several of our servers caused the automatic creation of Auto-discover records throughout all sites hosted on these servers.

These Auto-discover records were created regardless of whether the sites email is hosted with us or  hosted externally (Google Apps/Exchange); this caused pop-up messages to display to clients accessing email through our server asking them to update their email clients settings using the Auto-discover record, breaking their email.

We have rolled back these settings on one of our servers (dogfrog4/doc) to what they were set to before the upgrade and will roll back the settings on the other servers shortly; in the meantime we ask that all people still getting this message click on the ‘No” option and refuse to get there settings from the Auto-discover record.

This weekend, we’ll be replacing Snuppy / dogfrog5, one of our Australian servers, with newer, more powerful hardware.

While we are going out of our way to make this move invisible to you in every way, it’s possible that you may have temporary problems accessing your site on the morning after the move.  This note is just to let you know about that in case you are unlucky enough to be one of the few who experience a temporary issue.  You’ll be unable to access your site during the time of your move, for about 3-4 hours late on Friday night.  Details follow …

The migration of sites on Dogfrog5 will begin late in the evening of Friday the 18th of January, around 9:30pm. While we will do our best to make the transition as fast as possible, we expect it may take up to 6 hours. During the first part of the transition, your websites (including rotarydistrict9800.org) and emails will be unavailable.

During the move, please check this site for updates, and for any queries, as our main site will be down as part of the move.

Email will be queued during this transition; your email will be automatically kept and queued on the sending server and will come through to you during the next few hours after the server is restarted.

On Saturday morning, you’ll likely need to restart your modem first, then reboot all your PCs and other equipment before you can get your email again.

As always, we encourage you to make and download a backup of your site before this transition occurs. You can also get a backup copy of your site made and sent to you if you wish; please contact the office before Friday lunchtime if you would like to take advantage of this service.  (We encourage you to make regular backups although we do maintain an extensive backup cycle ourselves; the reason is just to cover yourself in the event of a problem with our backups).

For the very few of you who have your DNS hosted externally, it’s possible (though unlikely) that you may need to make some changes to your DNS zones, as the IP addresses associated with the new server will be different from the current one. If you think this will affect you, please call us on 1300 760 850 or email us at support@whitedoggreenfrog.com for advice.

If you are curious, this server has slightly faster disks, more memory (16Gb), a more powerful CPU, and a more recent version of the Centos operating system, amongst other things. It is located in Sydney, Australia in the Globalswitch datacentre there.  Our benchmarking suggests that in general terms, this server is about 50% faster than the old server and we expect it to respond to higher loads more stably.

If you have any queries, please call us on 1300 760 850 before 5pm on Friday, or email us at support@whitedoggreenfrog.com.

We will be on the phones on Saturday morning from 10:30am through 1pm to deal with any issues; and you can also email us at the above support email address from about 7am.

We do apologize for any impact on you or your business and we expect this upgrade to enable us to provide you with better service.

Regards,

the White Dog Green Frog team (soon to be WD3 team)

Overnight tonight, during the period from 11pm to about 3am, we are moving our Australian servers from one datacentre to another. The new datacentre, Global Switch in Sydney, is known as a top tier datacentre.

[Late Addition: dogfrog6.net is being moved tonight (18 Dec) from 11pm to 2am. Apologies for the interruption to service.]

We expect Australian servers to be completely unavailable for most of the above time period. The servers will be powered down at 11pm sharp.

There will be no change to IP addresses or software on the server and we expect the move to be invisible to those who are asleep during this time.

As usual, just a timely reminder to make and download backups of your cPanel accounts. We do make our own backups, which are housed both on the servers and remotely on other servers, but although we take every care and check regularly, we cannot completely guarantee those backups will work as expected.

The email delay that we had two days ago is back;  many new emails are getting through immediately but emails sent earlier in the day are facing up to 2 hours delay.

We’re looking into this; it looks like we’ve been hit with a huge amount of spam.  We’ll continue to monitor and watch this very closely until the queue drops.

Currently have a slight delay on email delivery on Snuppy (around 20 to 30 minutes approximately) due to high mail queues.

Update: Managed to get this under control late Monday night.

Dogfrog5 suffered some down time this morning due to a disk I/O problem which caused the main disk to go into lockdown (read-only) mode.

The server is up now and we’re keeping a careful eye on it. Apologies to anyone who was inconvenienced.

During the period the server was down, email will have been delayed.

Email sent to you during that period would have been queued on remote servers and is gradually catching up.  We don’t have any reason to suspect that any messages were lost.  It takes about an hour usually for remotely queued email to filter in, as the remote servers retry.

Websites and email should now be back up! It appears to have been effectively a minor, script based, Denial of Service attack.

We are watching the server closely to prevent a repeat of the incident.

Apologies to all affected.

Techs currently working on issue, websites may be up and down for a short period but other services such as mail should be up and running.

Dogfrog4.net is not responding to reboot or networks requests. We’re having technicians at the data center look at the issue now.

Apologies for any inconveniences this has caused you.