December 21, 2017 at 3:43 pm #4526Admin
Lvl 5 USTexas
Hey guys, sorry it’s been a couple weeks since hearing from me – My wife and I are expecting a baby this week so it’s been unusually hectic around here. Also what I’ve been working on is fairly boring to talk about, but I’m sure some of you just like to know I’m still here improving things. I said in my patch notes on the 8th that I’ll be working on server lag and that’s just what I’ve been focusing on.
It’s a technical topic and there are at least 5 different factors that combine to contribute to spikes and rubber banding. For example sometimes high server activity, population, dungeon generation, towers, botting, and general junk building up triggers a sort of memory cleanup. This cleanup can happen during low population times just to clean up junk from earlier high population times, so the amount of players online isn’t always a reliable indicator.
Population is only one factor. Since at this time I’m not able to afford geographically diverse dedicated servers I am using virtual private servers, which are a shared environment. There are going to be occasional peak traffic times where no matter how much I optimize there will still be bits of lag.
Optimizing different parts of my code to smooth out performance and cleanups should help. A couple weeks ago I started automatically cleaning up deep underworld floors during peak memory times. I have implemented some movement changes last week to reduce rubber banding. And for the last couple of days on USEast I have been testing a smoother object loop to reduce the duration of some spikes. I have more changes on the test server waiting to be pushed out. Unfortunately it can be hard to measure improvements from a player’s perspective since there may still be other lag causes present.
Another type of lag which is more framerate/graphical occurs on the client side. This can happen sometimes if the client is used for a long period of time. Closing and reopening the browser/tab should help in these cases.
I could go on about the other causes and types of lag and how I am targeting them but my time is probably better spent working on the problem rather than talking about it. On the topic of the new upcoming servers – they aren’t the cure for lag, or to give people a fresh start. They are mainly to satisfy demand from other continents who don’t have their own server yet. The highest amount of traffic after the US is from Brazil. New servers also help keep things less crowded, and of course have the nice benefit of reducing lag for a time. More North American servers should arrive as well, since the population is growing.
The occasional server restarts have not been triggered by me (and therefore I cannot give warnings for). It happens automatically when the server hits a memory ceiling and is usually due to factors outside the game server, for example a backup is being performed while leaderboards are being processed while underworld maps are being generated all at the same time. It makes it difficult for me to run tests sometimes. After I reduce the lag and have enough supporters across the additional servers I should be able to afford doubling the memory available on every server.
The backups are pretty critical and I would like to run them more regularly when I’m able. Recently a hacker/group from South Africa found a security hole in the database for USTexas and moved all the data to their database and held it for ransom, demanding Bitcoin for payment. Thankfully I had a backup from an hour before, restored it, and closed the security hole.
I recieved some messages expressing concern about the wall on USWest. There have been plans to add maps above and below market square that will connect to the snow and desert areas so the alternative routes should help when it’s added. These maps were planned before the wall on USWest showed up, but we’ll keep them simple for now so the wait isn’t too long. I don’t have an ETA just yet on that.
I’ve had a couple people tell me I should add more people to work on things so I’m not bogged down doing it all myself. I do have some people besides Kay that help me with various things which is helpful. As for development, the unfortunate reality is you can’t always throw more programmers at a project to make it go faster.. it can often end up slowing things down or fragmenting the project. I’ve tried this before in previous iterations of this game. Some developers introduced time consuming bugs and others even took the code to try to make their own idea of the game but it ended up failing and only hurting the community as a whole.
I think this is why indie developers are generally lone wolves while developing most of their project. It takes a lot of time and trust to build a skilled team of developers that mesh well together, and have the same vision. I would be happy to have a break and am open to outside help, but have to proceed cautiously for the sake of the community. With that said I have been reaching out and have made some tools for mappers to join and contribute. There might be forum mods added if needed, but in game moderators are not in the plans (it has been tried). It goes against the vision for the game and is too prone to bias and corruption. Other than that, I’m not too business savvy but if someone wanted to invest in the project and/or fund higher quality servers I could have some more flexibility with project development – but things will work out either way.
Some months I have more free time than others to get out content and features. In the past patches have been as frequent as once or twice a week. Though I do have to be a little more careful now with 200 active players compared to 20. The game has changed so much in the past year, it’s exciting to think about what it will be like next year.
Anyways I should get back to work. I’m still writing down bugs and features I see in the forums so there will be other things to look forward to other than just lag improvements. I’ll try to post these updates a little more regularly even during slow/boring development periods, just so you know I’m still working on things. Thanks to everyone who is supporting me and thanks for your patience with everything. Don’t forget to spend time with family during the holidays 😀December 29, 2017 at 2:46 pm #4611Admin
Lvl 5 USTexas
USEast, our oldest and largest server, is going to be down for a much needed upgrade sometime today/tonight. It’s the first of the servers to be upgraded, doubling it’s memory and therefore increasing stability. This also allows me to run important tests again to work towards improving lag and player capacity across all servers. The extra monthly costs of these upgrades are covered by your generous donations.
The website and game logins will also be going down for a few minutes sometime within the next week for general maintenance. Just a heads up in case you can’t log into the game or view the site for a bit.
Looking forward to a small patch soon. Have a Happy New Year!
The topic ‘Dev Notes December’ is closed to new replies.