22 July 2018
In this blog post, we will go over an ingenious system used in some of the older games created by id Software, in particular Quake 3: Arena, that made it very easy to create mods. The same system was used in games derived from the idTech 3 engine, such as Call of Duty 4: Modern Warfare. We will also discuss how you can implement it in your own game engine.
This blog post contains no new ideas, but this concept is so great that I felt like I had to discuss it.
The system is very simple, but very powerful. All game data is stored in ZIP archives.
At launch, you have for example a single archive, called dat0.zip. This archive contains the assets needed to run your game,
such as textures, sounds, levels, etcetera.
Now let's say you want to release some new levels in a patch. You could update dat0.zip, but then your users would
have to redownload the entire archive. No, let's create a new archive. We'll call it dat1.zip. This archive only contains
the new levels.
The requirements for your game engine right now are that it should scan all archives, and load data from all archives.
Nothing special, right? But here is where this system becomes really powerful. What if you want to release a HD texture pack?
It's very simple! Add another archive, call it dat2.zip, and fill it with HD textures with the same file names as the regular textures. Then have your engine always use the assets from the archive that has the greatest name alphabetically.
In other words, if a file exists in archive A and B, then the file should be loaded from B, because B comes last.
This system is very simple and has quite a bit of benefits:
Let's discuss that last point.
Let's say you want to make it possible for modders to create new game modes. Simple! Define your game modes in a script, package it up into an archive, and then modders can overwrite that same script file to add their own game mode.
With this method, it is also possible to create total conversion mods: modders can simply create a new archive, and your game executable will run the archive as is. Basically, your game engine becomes a simple tool that executes what is in the archive.
This system becomes very powerful if you use a scripting language, such as Lua, to drive your game logic!
If you do use a scripting language, make sure that it is properly sandboxed. You don't want users installing mods that can e.g. access the file system. The only thing that game scripts can do should be concerned with the game itself.
How do you implement this? It is actually very easy! Let's go over some code fragments that indicate how to implement such a system.
In these examples, we will use miniz, a great and small library for reading files from ZIP archives.
We will also ignore most error handling and parts that have no relation to the usage of miniz, such as creating a mapping from files to archives. The examples are written in ANSI C.
The first thing to do is to find the most up to date version of a file from an archive. We can do this by going over all of the ZIP archives in our data folder, and storing mappings from file names to archives.
DIR *dirFile = opendir(archive_path); // archive_path is the path to the directory containing all ZIP archives
if (dirFile != NULL)
{
struct dirent *handle;
while ((handle = readdir(dirFile)) != NULL)
{
// Ignore hidden files, current and parent directories
if (handle->d_name[0] == '.')
{
continue;
}
if (strstr(handle->d_name, ".zip"))
{
int i;
char *archive_absolute_path = malloc((strlen(handle->d_name) + strlen(archive_path) + 1) * sizeof(char));
// We have found an archive, now we will collect the files in this archive.
mz_zip_archive archive = {0};
sprintf(archive_absolute_path, "%s/%s", archive_path, handle->d_name);
mz_bool status = mz_zip_reader_init_file(&archive, archive_absolute_path, 0);
if (!status)
{
continue; // Handle error!
}
for (i = 0; i < (int)mz_zip_reader_get_num_files(&archive); i++)
{
mz_zip_archive_file_stat file_stat;
if (!mz_zip_reader_file_stat(&archive, i, &file_stat))
{
continue; // Handle error!
}
// We can't "open" a directory, so just skip them
if (!file_stat.m_is_directory)
{
// TODO: Create a new mapping if `archive_absolute_path` is greater than the current mapping (or if there is no current mapping).
}
}
mz_zip_reader_end(&archive);
free(archive_absolute_path);
}
}
}
Having collected these archives, you can now use them as a read-only file system. When your engine wishes to open a file,
it can use the mz_zip_reader_extract_file_to_heap function to extract the file contents as a void*.
First, you find the correct archive from the mapping, and then you simply call the above function.
void *contents = mz_zip_reader_extract_file_to_heap(&archive, filename, size, 0);
It's as easy as this!
What you do not want to do is reopen the archive every time you want to read a file. Decompressing files is much faster than accessing the file system, so you want to keep archives in memory for subsequent file reads. However, you should also make sure to not keep all archives open as once, as this will quickly fill your RAM. Monitor the memory used by the archives and close the least recently used (LRU) archives if needed. In short: handle open archives as a file system cache.
You might not want to use such a system for an open world game. Streaming assets is not really supported through this system as is, and it is better suited to games like Quake 3: Arena, where levels are clearly separated from one another.
This system is very simple to implement and still works after more than 20 years. You can patch easily, have good performance when loading your assets, and support mods all by implementing a system in one or two work days.
What are your thoughts? You can discuss this post on Reddit.