Recent searches
Search options
Administered by:
@theropologist@beige.party Attention Beige Party-goers!
In the past month there have been two incidents that I'm aware of where the web server was hanging and the site was unresponsive for several minutes. One of those seemed to resolve on its own eventually; the other I happened to notice as it was occurring and resolved once I gave the web server a hard reboot. It's possible that this has happened more than twice and I just haven't noticed it.
Looking at the logs I'm not seeing anything specifically failing, so I think what's happening is that occasionally traffic spikes are overwhelming the system resources. If left alone, the server can eventually recover but it will remain unresponsive until it does.
We have grown a lot in the last two years so this isn't totally surprising. I have scaled up the web server from time to time in order to account for the processing demands of increased traffic, but I'm reaching the limits of what I can do with a single server.
So, what I'm planning to do is spin up an additional web server and put them both behind a load balancer. This has a few advantages. If one server is overtaxed then traffic can be shifted to the other. It also means that for simple server updates (ones that don't involve database schema changes), I can take one down to run the update while the other stays up. In other words, in most cases I won't have to take the entire site down to do server maintenance.
This all sounds great, but as usual I have no idea what I'm doing. So I'm going to move slowly to set things up in the background and make sure I'm 100% certain everything is going to work before I push it live. Until I have this new solution in place it's possible things could be a little bumpy. So far the server hanging has been an intermittent issue, which is why I want to take care of it now, before it becomes a bigger problem.
As always, thank you for bearing with me!
Beige-bless 
@theropologist You’re awesome! We believe in you!
Also, whose turn is it to use the pillow tonight? I thought I had marked down Feb 8th.
@theropologist thank you oh mighty leader-ish!
@theropologist Thank you.
Patty. @theropologist 
Thank you for being utterly and reliably human in a world full of stupid glue-pizza topping Trump voting robots. So glad you are you.
You're the bomb.
Do kids still say that, anymore?
a mastodon admin myself, and just planning another instance -
would you mind sharing your server specs and number of users?
@sebastian Currently I have a web server that's running Puma, Sidekiq, Redis, and Elasticsearch, and then a separate server running the database behind pgbouncer. The web server has 16 cores and 32 gb ram. I'm noticing that at times most of that ram is getting used up. We've got about 650 users, with around 400 active within the last month. That's a lot for us, but not a huge amount in the grand scheme of things. I've decided that for the first step I'm going to move everything but the puma processes to a different server, so that the web server will only be functioning as a web sever. This might resolve the issue on its own, and if not it's still a good first step before introducing a load balancer. I'm also going to try tweaking my puma process/thread counts to something that's not as taxing on memory, but still provides adequate response times.
@theropologist thanks for taking care of business !!
@theropologist As a long time sysadmin, scaling and balancing cost is a hard problem. I appreciate the methodical approach.
Thank you for your efforts!
Beige-bless
@theropologist THANKS, MAN
@theropologist
We live for bumpy!
@theropologist We believe in you! 
@theropologist "but as usual I have no idea what I'm doing"
Love that!
@theropologist the leader is good, the leader is great!
It's me isn't it. You can just tell me I'm posting too much.
Antonius Marie 
@TheBreadmonkey @theropologist Checks out ;)
@amiserabilist @TheBreadmonkey
"Shall we make the keyboard more ergonomic?"
"No, remove hands. New hands"
@amiserabilist there aren't enough rampaging nude cyborgs in our cyberpunk timeline, another disappointment.
How's the translation thing going?
@ratcatcher I'm holding off on that for now because that's another process that could potentially further tax the web server. However, after reviewing my options this weekend, I've decided to move all non web server processes to a separate server. Once that's up and running and if things remain stable, I'll add the translation service to this secondary server so that it doesn't interfere with the web server itself.
@theropologist will one of the servers say "BEIGE IT" because i think the instance is lagging because the post box doesn't say "BEIGE IT"
we know that alone increases website uptime by 130%
@theropologist Really grateful for the work, thank you!
Also I appreciate your caution, it takes a big rooster to know your limits and tread lightly.
Awesome to have you as an admin!
@theropologist
Thanks so much for all the awesome you bring to the Beige corner of the universe .