Thank you all for the amazing first week!

We hit a peak of 3573 concurrent players (with 3805 connected clients) yesterday and during the week around 3000 concurrent players around 9 pm CET. Given the performance metrics, it looks like our player cap should be close to 5000.

On to some more technicalities and issues:
We started off with one freeze cause, one crash cause and somewhat regular lags that increase in duration the longer the server is up every 60ish minutes.
The crash cause has been fixed on Wednesday and since then we haven’t had a hard crash. The freeze cause has been found yesterday and a fix has been applied today, Sunday, time will tell if this fixed it for good or if there is another one lurking.

The regular lags are a different story and fixing them will be the focus next week. The first lag usually happens around 30 minutes after a server restart and takes around 1 – 2 seconds, the next one is usually after another 30 or 60 minutes and still takes 1 – 2 seconds and from there it happens around every 60 – 120 minutes and starts to increase in duration, after about 10 – 16 hours of uptime (depending on how many players are online) it reaches a point where the lag duration can potentially be more than 10 seconds which causes an disconnect for everyone which in turn leads to all sorts of fun that ends with a restart being required. 
The stop gap solution we’re doing until a proper fix has be done is to restart about every 8 hours, we’ll try to take posted raid schedules as well as spontaneous raids into account.

In case people care about the technical cause: We have a couple somewhat major memory leaks, npcs/mobs in particular, after a fresh restart the server uses around 25GB ram, after about 10 hours it’s up to 80GB and the pause itself is a stop the world gen 2 garbage collector pause.

Near the end of the week we also ran into a couple auth server issues caused by 3 different DDOS with one of them being rather persistent. This auth server issue should now be resolved but whenever a restart is required with a rather large player count we kind of run into a thundering herd problem, we’ll work on that next week as well.

Learn how to play and connect: https://playphoenix.online/