January 8, 2018 at 4:13 pm #4824
Hey guys, just an update on my progress. Most of my focus has been on lag and stability and I’m happy to report that it is much better now than before. There is still room for improvement but since things are running smoother I can focus on some other features for now. My new baby daughter is happy and healthy by the way, she tries to make sure I don’t get too much sleep.
So server stability was a big problem for a while, USEast had been going down every few hours because there wasn’t enough system memory to sustain the world. After the upgrade it was okay for a couple days but then started having crazy cpu usage, causing lag. Thankfully with the new memory ceiling I was able to profile things and get a report on what was causing the lag: Nothing. The code wasn’t showing any significant slow downs. Then I looked at the server console and realized that cycles were being stolen by neighbor processes on the same host. I put in a support ticket and they confirmed that our lag was being caused by noisy neighbors, and offered to migrate the server to a quieter host. After the migration the numbers have been awesome.. and the server hasn’t had a restart since.
Speaking of restarts and stability, the new Seasonal server and Brazil server have been amazing – not a single restart or high cpu reading since they were set up. I wasn’t able to say the same for USWest or Europe. They seemed to be competing for which server could become as unstable as USEast used to be. Europe started pulling ahead yesterday so I upgraded them first, and then 12 hours later did the same for USWest. While USW had great numbers it was looking like EU was having some cpu steal issues like USE had, not quite as bad but enough to be annoying. I wrote support again and moved the host earlier today so now all servers are looking good, though USTexas will eventually need some attention.
Aside from that I found and fixed a bug with the SK boss. The myst wasn’t being split properly among attackers. I changed the engine for towers so they ‘wake’ now instead of constantly checking which will slightly help cpu cycles. I increased decay rate very slightly, and reduced repair rates on structures. I’ve added a new mechanism for armor experience which stops awarding xp after so many hits from the same entity, which resets after a period of time. This will encourage more movement and active play. I fixed a rare bug with stairway creation, and a bug where dust wasn’t being awarded on reincarnation if you never died.
Well that’s most of it. I should remind everyone to please report any new bugs or exploits you find. It usually takes a while but there are definitely consequences for hiding exploits so be sure to give as much detail as you can, as soon as you can. Thanks!
Hoping to work on some more fun things soon. I’m seriously going to start reducing my non-mystera workload this year so I can do more for you guys. Have fun and see you next update!January 17, 2018 at 3:43 pm #4994
Hello! Here’s what I’ve been dealing with recently, related to development..
Have been working on Google Payments again. Something seems to be wrong with the payment library I’m using for Android so I need to find a way around that.
I fixed a bug with Chaos kills not rewarding assassin experience. Let me know if there is still anything wrong with that.
The datacenter I use has been forcing some downtime on the servers recently due to a weird CPU vulnerability. I’ve been trying to maintain the latest updates on that in the server maintenance thread until everything is resolved. There may be another couple restarts in store for everyone.. nothing I can about that but they are usually fairly quick (10-30 mins).
The population is growing again since the stability updates. We hit a new all time record for unique players online. There are 3 more servers in the pipeline for when things start to get too crowded. USEast and especially Europe have been growing a lot, so there may be new worlds added near those locations.
Keep in mind this game is not like most MMOs where it’s easy to support 200+ players on a server because only chats and simple movements are tracked. ML is unique because, as a sandbox, most of it is alive: grass and plants are growing, animals are breeding, buildings are decaying, the underworld is always changing, and almost every single NPC has a special behavior. There is a cost for simulating this world, so our communities have to be a little smaller compared to larger MMOs – especially on hardware that a free game can run on. I think it’s worth it to have a more dynamic world and no cash shops or pay-to-win elements.
There has been some talk about the decay changes from the last update, though I’m wondering if some of it is psychological. I didn’t actually change anything with the speed of decay, it still evenly targets buildings the same as before. The amount of HP taken off buildings during decay was increased slightly.. but only by about one percent (1.25% exactly). It’s not that much, and it’s been higher before.
The goal is for people to live within their means, not every new person can have their own mansion when they start.. it should take teamwork and effort to maintain a large amount of real estate. Otherwise we start to have a mess. With that said, this was the 7th time decay has been changed and it probably wont be the last. Same goes for repair rates, so you may see some buffs there again. It just always takes some time to see how things settle.
Server crashes are pretty rare these days. They were usually due to memory ceilings before I started doing the upgrades. It’s been nice to see consistently good numbers all across my server panel lately. I’ve identified one rare crash that is not memory related – it’s due to objects being removed during the new optimized loops, and has happened twice so far I believe. It’s rare enough that I haven’t dedicated much time to it just yet.
The other issue I’ve noticed is a rare problem where the garbage collector can get bogged down for a period of time. This doesn’t happen often but it can cause lag for 30-90 minutes before it fixes itself (unless I notice it and force a restart before then). Feel free to send me a contact form submission if it’s happening to your server.
I will revisit lag & stability again at some point, after I’ve finished some other things. Though sometimes lag is just lag – if you have a high ping to the server due to your location or connection then you will get occasional jumps.
A couple new people have sent me reports that they can’t connect to any servers, they just press a world button and it kicks them back out.. On both the website and their phone. This makes me wonder if they have a firewall or port blocked on their network.. I’m not sure what else that would be at this point.
So that’s the state of things this week. Very exciting, I know. Well it is my birthday in a couple days, but don’t worry I’m in my thirties so I wont take the day off or anything 😛 I’ll be working on improving some things on the client when I get a chance and there are some new maps, NPCs, and cosmetics to look forward to.January 27, 2018 at 2:55 pm #5107
Just a short update to let you guys know I’m still here and working on things. It’s mostly dry and technical so feel free to escape lantern out now.
I did make some progress on Google payments, and was able to get past the issues with 3rd party library I was using. I had started running into some hurdles with payment validation when a higher priority problem came up. So I paused work on the client to debug some potential memory issues in the server code.
I mentioned in my last notes that there was an occasional problem where the cpu would jump up really high and stay there for a while, usually requiring a manual reset to fix the resulting lag. That was a rare issue, until this week where it started happening regularly to the large servers, after 40 or so hours of uptime. This led me to pull apart a lot of my code looking for leaks or potential memory optimizations. I’m not sure that there were any leaks but I’ve been able to clean things up some, optimize memory usage, and even speed up the networking loop so we will see. I’m not ready to make that code live yet but it should be ready soon. Then hopefully the cpu issue will either be gone, reduced, or I can narrow down the problem better. At least until then I’ve set up better automatic notifications when there are cpu issues so I can restart sooner when this happens.
There is also a new noisy neighbor on USTexas recently – the server cluster is stealing some CPU cycles from us again which has been a source of lag in the past for other servers. If it continues much longer or gets worse I’ll put in another support request to the datacenter asking to migrate to a quieter spot. If only there were cheap dedicated servers with multiple locations around the globe..
That’s about it for this week. Sorry I haven’t always been able to reply to support emails due to lack of time. I’m mostly responding to critical issues. Oh I’ve seen some of the new cosmetics that are on the way, that will be nice 🙂
The topic ‘Dev Notes January’ is closed to new replies.