Welcome back to another instalment of the Blightmare developer blog. We’re happy you’re here! Today’s post will walk through the server support that we built to enable our editor to save, share, and protect levels. Let’s dive right in.
As we’ve talked about before, Blightmare has a level editor application that is used to create all the playable content in the game ranging from placing the terrain and mechanics, to creating simple scripts for curated interactions, to actually defining what order levels come in when playing the game. All of this data needs to be stored somewhere safe and shared with all members of the team so that we can test and iterate on things. We also need to make sure that we’re not losing any work by having multiple people creating conflicting changes to a level. I’ll first start by giving an overview of how our data is organized, then I’ll talk about how we implement the services and what technology we use, and finally I’ll revisit the multiple changes problem.
Starting with the data is always a good way to begin when tackling a problem, and it’s no different here. Blightmare has a pretty simple set of data types that are managed by the server:
- Level Versions – A level version is the actual data that the game loads when a level is started. This is currently a blob of JSON, but we may perform some further transformation on the data when we package up a game for distribution. A level version is immutable which means that once it is created, it never changes. Level versions are identified by a randomly generated identifier (called a GUID) and are the most basic data type in the model.
- Levels – These are the things that someone gives a name to and we usually talk about. Levels can be contained in folders and can have many different versions which track the history of the level over time.
- Level Lists – Levels are collected into a list to define a whole game experience. Right now, Blightmare has 17 entries in the “Whole Game” list and there are several other lists for various testing purposes. Eventually the Level List structure will become more than just a simple list because some levels circle back on each other, and there may be secret or bonus levels that are kind of like offshoots from the main game path.
- Folders – We’ve created many levels so far while building Blightmare. Over 100 actually, and in order to keep things organized, we put them in folders. Folders can be stored in folders as well to create a nice neat structure.
There are a number of operations that we want to be able to do with this data and most of it won’t be surprising. Blightmare is actually pretty simple in this regard and doesn’t have much more than the standard Create, Read, Update, and Delete. Some of these are named differently to make more sense in the context of our model, and we combine some operations together to make things a bit easier. For example, saving a level is very common, so it is offered as a single operation but it’s composed of creating a new level version, then updating the level to point at it. We have some search functionality added to our server’s interface as well as some general management functions to assist with backups.
The server itself is written in Java and uses the Jetty server toolkit to make things easy. We use the excellent Jackson library to convert Java objects into JSON for transport over the internet. We use what is called “embedded” Jetty which means that we are a bit more manual with the setup of how Jetty interacts with our code than some other setups. This avoids most of the XML configuration and makes things a bit easier to wrangle in my opinion. It means that we don’t need to worry about building a WAR or any other magic bundle of things. Instead, we build a pretty generic runnable JAR, copy it up to our server, and run it.
Storing all our data becomes another issue that we had to tackle. With over 100 levels, you can imagine that there’s many hundreds of versions of those levels since we keep every version of every level. Combine that with all the data storing which version is associated with which level, where they are in the folder structure, and which version is referenced by a level list, there’s a bunch of things to keep straight. We use a pair of tools to help us here: Amazon S3 and Amazon DynamoDB. The level version data itself is stored in S3 and is referenced by the GUID that I mentioned earlier. All the rest of the data is stored in DynamoDB. DynamoDB is a no-SQL database that is offered as a managed service where you only pay for operations on the data. We use a couple of tables within DynamoDB to store all the references between our data, and we use some well-known names to make sure we can find our data quickly. For example, the root of our folder structure is in a folder with the special id “00000000-0000-0000-0000-000000000000”. From this folder, all other folders and levels can be found. The best part about the AWS services is that for our usage level, it ends up being less than $1 per month.
Now that you have a high level overview of how the basics of the server work, I’ll finish up by addressing the multiple-editor problem. The way that our system is setup means that a local editor instance loads up the data for a level and does a bunch of edits to it while a designer is making changes. When that’s complete and the designer wants to save, the editor creates a JSON representation of the whole level and sends it up to the server. This means that only the changes that were made in that editor since the last time it loaded the data will be stored. If someone else had saved their changes to the same level in between, then those changes would be overwritten. This is obviously not something that we want to have happen, and it’s even worse because there’s no warning. We could use something like Jira or our Discord to communicate to each other about who is using what, but this is very brittle and error prone. Not to mention that this is a perfect job for a computer!
There are a couple of ways we can solve the problem. The fanciest way would be to build something similar to google docs where multiple people can edit a level at the same time and the changes are seen everywhere in real time. Typically fanciest also means most complicated and that’s definitely true here. It’s quite a bit of work to build something like that, and it would require significantly more server resources to execute. It’s also complete overkill for how our team works. We almost never have work happening on the same level by multiple people, so the solution should reflect that. What we went with is a simple locking scheme that the server enforces. When the editor starts up, it asks the user for a name which is then used to lock levels for edit. Locks are only needed when editing a level – playing them isn’t changing data so it’s safe to play a locked level. The UI can now show which levels are locked by you, which levels are locked by others, and which levels are unlocked. Additionally, locks can be released at any time through the UI. The server requires that a token be sent with a level version request which it uses to verify that the change is being made by the lock holder, and that way we prevent anyone from making a change when there’s a chance of conflict. The editor client also ensures that the user holds a lock for a level when trying to open it to make sure we don’t have any confusing situations.
Locks are stored on a file on the server so they persist across server restarts, but are also very cheap and easy to inspect in case something goes wrong with a lock, or someone goes on vacation while holding a lock. This solution is quite simplistic and would not hold up in many other more complicated scenarios, but for our simple use case it has prevented any loss of work without significant cost to workflow or implementation time, so I think it’s worked out great.
That brings me to the end of this overview of our editor server and supporting tools, I hope you enjoyed it! If you like this content and want to check out the game, please head over to Steam and add Blightmare to your wishlist. To make sure you don’t miss an update about the blog or any other game news, follow us on Twitter. Stay safe and have a great week. Until next time!