A message in the morning.
As some of you may have noted in the last few days our little website was having a (few) bad hair day(s). I was blissfully unaware of the havoc that was going on until I woke up on Saturday morning with a message from Stefan asking if I also had some dreaded 503 service disconnections the last few days.
A rogue coffee cup.
One coffee cup later, and I could confirm that yep, I also got a 503 service disconnection, and would take a quick look at it. What better way to start a Saturday morning than saving the day and rescuing our little blog from a plugin gone rogue.
What started of as a simple item task, soon became slightly more complex as I could not connect, and even the backend of our little web server was timing out.
Looking at our CPU (that egg cooking thing that actually does all the hard work) things looked even worse.
Since May 9th we seem to have been under constant attack.
Who or what could be so evil?
We had a few extra spam messages of late, but nothing major.
And checking our Cloudflare instance, no major attacks were detected or highlighted.
Who you gonna call?
Time to call in the good folks of Siteground (our dedicated web hosting service we warmly recommend) and ask them for expert assistance.
The first line chat support routed me directly to an senior expert, and this lady pointed me out to some heavy loads on our options table. If you ever mingled in the dungeons of your WordPress blog, you know the wp-options table can be a killer and indeed that was number one on the list.
A good hour later, I cleaned up quite some smashing balloons orphaned entries (the reason why our IG plugin is down) and brought the table size back from a whopping 200MB to a good 5MB. Good job done.
But the problem was not solved.
The CPU kept cooking eggs and our 503 service disruptions did not get any better.
The good folks of Social Warfare sent us a mail earlier in the week that they had a major upgrade, and it was not the first time they created havoc on our little site.
Disabled as well, and the jury is still out if we bring these beautiful social media icons back or not. Let us know in the comments.
I personally love them, but it is not that we are a powerhouse of social sharing and they tend to have a mind of their own.
Still not fixed.
Disabled a few more plugins.
Still not fixed.
Back to the good folks at support but no big haha moment either.
Heavy queries are considered to be the culprit of the 503.
Not a squadron of stormtroopers, but an exhausted webserver trying to cope with the load.
Maybe it is a squadron of bots after all.
Looking at the last visitors, and the spam mail we got earlier in the week, time to up the ante and get out the credit card.
Upgrading our Wordfence plugin to a state of the art “premium” edition with real time blacklists.
That should stop the bots and stormtroopers alike.
Except I cannot install the new premium API key.
And since it is a binary in the database, I go ahead and remove the full plugin including all its data and do a clean fresh reinstall.
It accepts the new key, and goes into learning mode.
Alice in Wonderland.
And here we get into Kafka, euhh, Alice in Wonderland.
Our service provider says the problem is now caused by … Wordfence.
Some heated back and forth emails and I hand over the keys to the fortress giving them full reign to solve the riddle.
It is getting late here in Stockholm and I go to bed frustrated.
A full day has gone by, and no real progress.
Sunday morning, 4 AM.
The sun is up, the birds are chirping and I have a new notification in my inbox.
The root cause of the problem is caused by a Cloudflare extension, called Rocket Launcher that is creating a massive load on the server by wrongly executing a Wordfence command that is cached in its server.
Basically one click on a link triggers a thousand calls in less then a second, causing our little server to throw the towel in the ring.
I log on to Cloudflare to disable the Rocket Launcher and cannot stop smiling when I see another feature in their list.
A few clicks later and clearing some secret caches of hidden web files and all is good.
Time for some coffee after all.
PS. Not your classic LEGO post. Just a fun little behind the scenes of running a blog and a tribute to all those wonderful people that keep on developing awesome stuff.
PPS. If you are a die hard visitor of our site, you may still have some of the old stuff in your browser cache, and clearing the browser cache will fix the issue for the time out.