-
Posts
-
Quite a busy week for me. While investigating and optimizing the memory/stability issues on the server I just decided to rewrite everything. I spent all week optimizing hundreds of items, effects and npcs. They are now way more memory efficient. This wont always be apparent in your immediate play but it should make a difference on a bigger scale. I slowly applied these changes to all servers at different times yesterday.
So before when a server started up the memory was already at 60% of its ceiling, eventually growing to meet it after 24-44 hours. That then bogged down the CPU as it desperately tried to manage it, which caused those periods of crazy lag you’ve all experienced – requiring me to manually restart.
Now when a server starts up the memory is at 30% of the ceiling, and so far it seems to stay contained there. Even USEast with over 150k items laying around now only takes about 600mb RAM and it doesn’t seem to be growing. For you this means less restarts overall, and hopefully no more periods of insane lag.
This has had an effect on the CPU too it seems, as I’m seeing better numbers across the board. So that will translate into slightly less lag and smoother gameplay for everyone (as long as a neighbor server doesn’t start stealing our CPU cycles again). You should also see less disconnects due to packet flooding or normal internet lag.
So we may see our player ceiling go up, and I will still be adding new servers. On that note the datacenter that Brazil is on has been disappointing, it has had multiple periods of downtime for no reason and it is costing over 7 times what the other server company I use charges. So I will be migrating it to another hosting company in the region as soon as I get a chance.
Anyways there are still plenty of areas to improve with the game, but we’re making good progress! I want to get back to putting out some content for you guys and quality of life features and bug fixes. I will also be working on Google payments and dealing with some exploit abuse. Thanks for your patience and support!
There are a few nice changes in this patch, which I just made live. I’ll probably get a front page post up soon with a summary of these last dev notes, even though it’s not as much as usual due to being mostly backend work.
Before if you wanted to build underground it required you to have a multilevel base because you needed to secure an upper level to prevent people from building stairs down into your lower level base. That is no longer the case – now when you have a floor underground it effectively makes a ‘ceiling’ that will block stairs from being placed. So now you can build completely underground homes and cities, freeing up space above ground. This will make hiding bases a little easier as well. You can still build floors under a stairway after it has been placed, but as a precaution I also made it so escape ropes can no longer rope onto a lower stairway. The max hp of stairs will be set to 10k instead of 1k from now on.
The infamous stuck-on-the-stairs bug should now be fixed. Let me know if you notice any other problems with that. I fixed a bug with /unstuck and an exploit that let people leave newbie village early. Also made rafts stop floating around after so many minutes of nothing happening.
Digging up road and floor tiles has been made more difficult, or more easy, as it depends largely on your digging level now. Also the type of floor or road makes a big difference. If you have a low digging level you may be able to dig up a clay floor before too long, but a stone floor could take several minutes. Stone road is even harder to dig up and it may be worth recruiting someone who has a high digging level to help with big projects – otherwise you could be using a lot of time, and shovels. Will we see a better shovel soon? Maybe! In any case this should reduce the impact of trolls who dig up roads for fun.
In other news I can share what I’ve been doing in the background for a couple months as related to exploits. I developed a tracking system that shows me a lot of data about what certain suspicious individuals are up to – so when I receive a report on potential exploit abuse I can start collecting evidence. I understand some people are frustrated when others are ‘allowed’ to do these tricks in game – but sometimes if the exploits don’t affect balance much it’s better for the game as a whole if I just observe and collect data for a while.
A few people have provided a LOT of data on exploits, and I wanted to wait until they were finished poking around, without spooking them too much. If I just start punishing people right away for finding small exploits then I miss out on them finding bigger exploits as they gain confidence. I’d rather have the help discovering those critical bugs now, than later on when we may have many more players and servers to manage. So I appreciate everyone’s reports and patience during this process, it has been very fruitful for the health of the server code and I hope you can understand why I was trying to stay somewhat quiet about these things for a while.
So now I have enough data and evidence, a lot of exploits have been patched, and in addition to my tracking system I have just finished developing a punishment system as well. In the past when there was just one server there was a tribe who abused some severe exploits, like killing people who were sleeping in their own houses. The specifics of that bug were unknown for quite some time. After I finally had my evidence all in order (which was months after the abuse) I suspended them for weeks. Suspension isn’t always the best tool though as there are ways to get around it. Some people don’t mind making new characters and using new IPs. So now I have another more painful tool that lets me quickly and easily reset an aspect of a character’s progress whether they are online or not. I can easily apply it to alts as well, on any server. These aspects include skill experience(levels), myst, angel dust, upgrades, and even blue stars.
Before I talk about who is receiving this I want to go over the ‘one rule’ again. This is the only in-game rule on the website that gets shared and passed around so people will be well aware of it: If you find an exploit then report it, don’t abuse it. It’s common sense in any online beta. The act of finding exploits is very helpful – and in some cases I’ve even awarded diamonds for receiving specifics on reproducing them (as long as they weren’t abused). If you don’t report those specifics right away though then it is very unhelpful because it wastes my time looking through logs and trying to reproduce things on my test server. Also if you are a friend or tribemate who learns about the exploit abuse (especially if you know anything about how it’s performed) then even if you don’t use it yourself you may get a punishment just for hiding information from the developer. When in doubt send a contact form with what you know.
Some people use simple scripts to perform repeated actions or look up data provided to the client like coordinates – that is generally okay. If you start manipulating network packets or performing a certain action to exploit a bug in the server code that allows you to do something in a way you shouldn’t be able to then that obviously isn’t okay – especially if it affects balance or hurts other players. So basically feel free to find exploits but be sure to report everything about them – otherwise you will eventually be tracked whether you know it or not, and could lose a lot of progress in the future making your efforts wasted.
I still have various people being tracked (some who are not listed below), and I make sure to verify with server logs that they are indeed exploiting or hiding information about exploits from me. The people who were demanding justice right away will at least be comforted to know that the longer these punishments were delayed the more severe they became as it takes time to build these things up. I wont hesitate to repeat or increase these punishments as necessary.
The current players with sufficient evidence against them for abusing or hiding exploits:
Krill and his notable alts: Loss of all levels/skills, myst, upgrades, dust.
Lodni and his notable alts: Loss of all levels/skills, dust.
Therieth, Badboy, Fluffylass: Loss of dust.
Edit: Added Belial: Loss of dust for breaking forum rules repeatedly and circumventing the IP ban (this will become more severe if it continues).This applies across all servers. I hope in the future if another exploit is found I will be receiving reports and possibly even awarding diamonds, instead of handing out punishments. It will be nice not to have to waste time on this so I can focus on other things 🙂
Now back to work on content, features, and bug fixes. By the way, the numbers on the servers have still been pretty awesome compared to last week, the high cpu lag bug hasn’t happened once so I haven’t had to restart at all. There is still 1 rare server crash that I need to fix, which has happened a couple times since my last notes. I’m definitely keeping an eye out for new bugs as a result of the rewrite. Also even though the lag is better since the memory optimization it is only one aspect of the various factors that cause lag so I still have more work to do to reduce lag further. Now go make an underground base and report some exploits! 🙂
-
This reply was modified 6 years, 2 months ago by
Admin.
Just a quick update on the testing and changes that I’ve been making in the background:
In the server code I replaced the networking library with one that is better maintained. It may help with lag and stability some, time will tell. All servers have been restarted to test it.
Fixed an issue some were having, where the loading bar for the game would pause or get stuck.
You can no longer get infamy if your character is sleep-attacking.
USEast needed some cleanup of underworld maps, so I did that yesterday and also changed shovels so they can’t dig beyond floor 100 for now. The cleanup is mostly automatic but I need to optimize it more so I’ll be making some more UW adjustments when I have time.
Now that underground bases are a thing we were seeing the random spawn-teleporting problem more often, on login/restarts. I changed how this works: if the server can’t find a nearby nature tile to place you on then it will try to place you on your recall tile, if it can’t do that then it will place you at crossroads.
I also made it so you can go up and down stairs even if someone is standing on them. To prevent player stacking abuse on recall tiles or stairs I made it so you can’t attack effectively if you are on the same tile as another player.
There’s some other behind the scenes work but that’s about it for now. Let me know if you find any new issues or bugs. I’ve already handed out a few diamonds for more serious reports. I added something to the contact form that saves all submissions in the database in case the email fails for any reason, so feel free to use that. You can also PM me or Kaylovespie on the forums to report things.
I’m going to shift to working on some client side issues soon (for instance some crash/lag bugs with gaining infamy among other things) and prepare for a new patch 🙂
Hey all – another status update on the game. So I’ve been obsessing over the periodic lag issue these days, more than I should probably. It’s the slight rubberbanding you notice every so often, and gets worse over days of uptime. It’s what led me to rewrite most the server, and completely replace the database and network libraries. All these big changes help in a lot of ways but don’t cure the problem.
I seem to be running into some limitations of the underlying language itself at this point – where their garbage collection system is inadequate to handle all of our ‘junk’ over time. So I either have to implement my own low level memory system completely or wait for some updates the google javascript guys have coming. One of those updates will let me utilize multiple CPU cores for memory collection for example.
I added a new addition to my server panel this week that shows me the periodic ‘delay’ of each server, in addition to the cpu load. So as a result of my optimizations over the last couple of the months the cpu load, memory, and uptime don’t beep at me much anymore.. the numbers have been great. Now that I’ve added delay it’s been beeping at me again, so I can also feel your rubber-banding pain.
I’m thinking that I shouldn’t slow down content and bug fixes over this issue any longer. I’ll just slowly optimize it when I have time and restart the servers every now and then to reduce the pauses until I get a better system in place. It seems people don’t complain much until the servers have been running for some time anyways.
Another thing I added recently is a rapid system for sending my code changes to all servers and making them live instantly. It used to be a tedious process on each server. This convenience can be a double edged sword sometimes.. Normally for big changes I do phases of testing on different servers to see the effects. This week I had tested some little optimizations on my local server and then quickly sent them out to all servers which seemed fine for a while. Eventually I discovered that the spawns were not being blocked by players any longer – that was fixed easily enough (though it needs to be applied to some servers still). More critically all new players in the last couple of days were saving multiple versions of their account to the database.. which sometimes resulted in loading an older version of themselves. While established players weren’t affected, some new players were loading into a level 0 version of themselves instead of level 10 or so.. which I can only imagine was very confusing. I had to stay up late at night fixing that one, and cleaning up all the extra database records on each server. I left the highest level account I found for each player, so hopefully that’s all cleared up now. Thanks for the reports on these things by the way!
Another bug I’ve been working on, and may be fixed now, is the one where you log in and your client freezes up. Requiring you to relog again to see your character. This was a networking issue related to when you are moved off of floor tiles when you log in. I fixed all the cases of this that I know of, let me know if you still experience it.
So I need to move forward on some of my big plans and will need to devise ways to lighten my load if I’m going to ever defeat this todo list. One thing I’m implementing is like the multi-server code propagation feature I mentioned above, but applying to maps. So quick changes and additions to maps can be made live on all servers instantly. It’s a real chore at the moment. I’m also going to make a mapping server to invite people (on a trial basis) to create and test new areas for the game. If you are creative and interested feel free to send us a message about it through PMs or the contact form, though we may not respond to all requests as we can only train so many mappers at a time.
I am also having someone look into adjusting some numbers and balance for me related to equipment and npcs. Bias has always been a problem with this area but I will be supervising and approving any changes. The upcoming balance changes will hopefully be in time for the 2 year birthday of Mystera Legacy which is coming up fast (last year we got the iron age..). Though you will see new mobs before then – hopefully this week!
Anyways I’m excited to dust off the client and implement some much needed bug fixes and cosmetics soon. I apologize for all the issues and delays. I just want to make sure we have a really good core engine to build all this cool stuff on. Thanks for your testing and support.
-
This reply was modified 6 years, 2 months ago by
The topic ‘Dev Notes February’ is closed to new replies.