New tricks for old MUDs: using JSON for flexible player data

In the beginning (1990), the creators of Diku had many programming challenges.  One of the very least interesting of these was how player data would be written to and read from permanent disk storage.  This was before the days of structured data formats and object-oriented databases – some subset of Staerfeldt, Madsen, Nyboe, Seifert, and Hammer had to roll their own format.  Sometimes I wonder if they thought that their work would still be around nearly 30 years later.

As it turns out, the original method they came up with for storing players was a little terrible.  Diku stored all players in one binary file, and all of their possessions in another binary file.  Altering either of these would corrupt your entire player base. A year later Merc 1.0 would take a big leap to fix this, by storing players (with their stuff) in individual binary files.  You still couldn’t modify them unless you were handy with a hex editor, but at least only one player could be messed up at a time.

The Almighty pFile

It was Merc 2.0 (1992) that would introduce the format that most of the Merc derivatives still use: the human readable “pfile.”  For the first time, game administrators could search and modify the files as needed.  Here’s an example, straight from a fresh save on Merc:

#PLAYER
Name         Montrey~
Description  Some guy is standing here.~
Sex          0
Class        1
Race         1
Level        10
HpManaMove   20 20 100 100 100 100

What we see here are strings terminated by a tilde character, simple integers for Sex, Class, Race, and Level, and an array of integers for HpManaMove.  The reading behavior is defined for each key – whether to expect a string, int, array, or something else.

This was a huge improvement over previous games – and not just because the admin could tinker with it.  Merc 2 introduced treating the file as a key-value store, where reading the key (such as “Name”) could be mapped to the correct location to store the value (“Montrey”).  Keys could come in any order as well, to prevent additions or removals from making files unreadable.  This was accomplished with a nifty macro to hide a huge ugly bunch of “if” conditions:

#define KEY( literal, field, value )        \
    if ( !str_cmp( word, literal ) ) {      \
        field  = value;                     \
        fMatch = TRUE;                      \
        break;                              \
    }

void fread_char(CHAR_DATA *ch, FILE *fp) {
    for ( ; ; ) {
        char *word = feof( fp ) ? "End" : fread_word( fp );
        bool fMatch = FALSE;

        switch ( UPPER(word[0]) ) {
        case 'A':
            KEY( "Act",         ch->act,          fread_number( fp ) );
            KEY( "AffectedBy",  ch->affected_by,  fread_number( fp ) );
            KEY( "Alignment",   ch->alignment,    fread_number( fp ) );
            KEY( "Armor",       ch->armor,        fread_number( fp ) );
    ...

The inflexible pFile

If you’ve ever worked on a Merc derivative (like ROM), you’ve seen all this in save.c.  It works well, to an extent.  Why would anyone want to change it?

Rather than list all the abstract reasons why a portable, standardized structured file format is better than the Merc standard, I’ll describe the use case that led me to the decision to change.  We at Legacy are planning an equipment enhancement system based on placing “gem” objects in a piece of equipment to add stat boosts.  Simple enough, I thought.  I’ll just make any piece of equipment able to contain a list of “gems”, add up and cache their effects, and done.  It was all clear-cut… until I faced save.c.

In Merc muds, objects (of the container type) can hold a list of other objects.  When the game saves the player’s inventory, it writes the first object, followed by the first thing (if any) that it contains, writing the nest level for each object so that they can be read correctly in reverse order.  (Actually, because of how singly linked lists work, it uses a recursive function to write the list in reverse order, so that it can be loaded in the correct order without having to iterate to the tail of the list.  But that’s not important.)  The pfile ends up looking like this:

// inventory of "bag" containing a "potion", and then a "sword"

#O
Vnum 3716 // potion
Nest 1
Wear 0
Cost 2000
Val  0 0 199 0 0
End

#O
Vnum 3703 // bag
Nest 0
Wear 0
Cost 500
End

#O
Vnum 3704 // sword
Nest 0
Wear 11
Cost 10000
End

So how to add a new savable list of objects that can be “contained” in another object? What if you put gems in a bag?  How about a bag encrusted with gems?  Hrm.  You could write one list after another… but then you’d have to remember the nest level for “gems” and for “contents.”  You could come up with all kinds of convoluted ways that would make the code harder to read, harder to maintain, and easier to break.  The limitations of the pfile format were starting to become obvious.  The format was meant to store sets of values, but only barely supports nested objects.  I wanted a more elegant solution.

Enter JSON

JSON (JavaScript Object Notation) is a fairly modern structured storage format, similar to XML without all the <>>; nonsense.  A JSON file can store 3 types of things:

• values (strings, integers, floats, etc)
• lists (of values, like a C array), separated by commas, enclosed within square brackets ([])
• maps (a set of named values, like a C struct), with keys : value pairs separated by commas and enclosed within braces ({})

Most importantly for my purposes, these can be nested – you can store, say, an array of maps.  Some examples can be found at json.org, but it might be easier to just show a Character expressed in JSON:

{
  "Character" : {
    "Name": "Montrey",
    "Description": "Some guy is standing here.",
    "Sex": 0,
    "Class": 1,
    "Race": 1,
    "Level": 10,
    "HpManaMove": [20, 20, 100, 100, 100, 100]
  },
  "Inventory" : [                  // list of objects (maps)
    {                              // start of "bag" object
      "Vnum": 3703,
      "Wear": 0,
      "Cost": 500,
      "Contents" : [               // list of objects (maps)
        {                          // start of "potion" object
          "Vnum": 3716,
          "Wear": 0,
          "Cost": 2000,
          "Val": [0, 0, 199, 0, 0] // list of ints
        }                          // end of "potion"
      ],                           // end of "Contents"
      "Gems" : [                   // start of "Gems"
        {                          // start of "gem" object
          "Vnum": 4000,
          "Wear": 0
        }                          // end of "gem"
      ]                            // end of "Gems"
    },                             // end of "bag"
    {
      "Vnum": 3704, // sword
      "Wear": 11,
      "Cost": 10000,
    }
  ],                               // end of "Inventory"
  ...
}

The root of the JSON file is a map, with named values contained in it.  Name, Description, Sex, Class, Race, and Level are string or integer values.  HpManaMove is a list of integer values.  Inventory is nested map of the top-level inventory objects, with Contents and Gems optional lists of more objects.  Obviously, you probably don’t want to write complex JSON files by hand, but it is still pretty straightforward if you need to occasionally grep, sed, or fix something in a text editor.

Serializing MUD objects with cJSON

So how do we make it happen?  I won’t lie, it did take a while to convert Legacy to use JSON for pfiles, but once I muddled my way through it, it turned out to be pretty simple.  I decided to use the cJSON project, because it is lightweight (just a single include and source file) and had the reasonably simple semantics I wanted.  Here’s a shortened example of writing a character to file:

void save_char_obj(Character *ch) {
  cJSON *root = cJSON_CreateObject();
  cJSON *char_cJSON_obj = fwrite_char(ch);
  cJSON *inventory_cJSON_obj = fwrite_objects(ch, ch->carrying, FALSE));

  cJSON_AddItemToObject(root, "character", char_cJSON_obj);
  cJSON_AddItemToObject(root, "inventory", inventory_cJSON_obj);

  char *JSONstring = cJSON_Print(root);

  char strsave[MAX_STRING_LENGTH];
  strcpy(strsave, PLAYER_DIR);
  strcat(strsave, ch->name);

  FILE *fp = fopen(strsave, "w");
  if (fp != nullptr) {
    fputs(JSONstring, fp);
    fclose(fp);
  }

  cJSON_Delete(root);
}

In this code, we simply create a root object (which is a JSON map) and add some named objects to it.  Then we let cJSON convert the whole thing to a string and write it to the file.  cJSON takes care of the parsing for us too, so at the high level, reading a pfile is essentially the reverse:

Character *load_char_obj(const char *name) {
  Character *ch = new_char();

  char strsave[MAX_INPUT_LENGTH];
  strcpy(strsave, PLAYER_DIR);
  strcat(strsave, name);

  cJSON *root = JSON::read_file(strsave);

  if (root != nullptr) {
    fread_char(ch, cJSON_GetObjectItem(root, "character"));
    fread_inventory(ch, cJSON_GetObjectItem(root, "inventory"));

    cJSON_Delete(root); // finished with it
  }

  return ch;
}

Constructing and iterating through cJSON objects

Of course, the real meat of it in the fread/write_char and inventory functions.  Here is how the “character” section is written – actually not that different from the default fprintf statements:

cJSON *fwrite_char(Character *ch) {
  cJSON *item; // reusable pointer to save lines
  cJSON *o = cJSON_CreateObject(); // object to return

  // add some regular values
  cJSON_AddNumberToObject(o, "Act", ch->act_flags);
  cJSON_AddNumberToObject(o, "Alig", ch->alignment);

  // not all values are required
  if (ch->clan)
    JSON::addStringToObject(o, "Clan", ch->clan->name);

  // how about a list of simple values?
  item = cJSON_CreateObject(); // create a new item to add
  cJSON_AddNumberToObject(item, "str", ch->stats[APPLY_STR]);
  cJSON_AddNumberToObject(item, "int", ch->stats[APPLY_INT]);
  cJSON_AddNumberToObject(item, "wis", ch->stats[APPLY_WIS]);
  cJSON_AddNumberToObject(item, "dox", ch->stats[APPLY_DEX]);
  cJSON_AddNumberToObject(item, "con", ch->stats[APPLY_CON]);
  cJSON_AddNumberToObject(item, "car", ch->stats[APPLY_CHR]);
  cJSON_AddItemToObject(o, "Atrib", item); // add whole array to o

  // let's add a list of maps
  item = nullptr; // could be empty, only build if there is at least one
  for (const Affect *paf = ch->affects; paf != nullptr; paf = paf->next) {
    if (item == nullptr) // got one, initialize
      item = cJSON_CreateArray();

    cJSON *aff = cJSON_CreateObject(); // build a new object
    cJSON_AddStringToObject(aff, "name", skill_table[paf->type].name);
    cJSON_AddNumberToObject(aff, "where", paf->where);
    cJSON_AddNumberToObject(aff, "level", paf->level);
    cJSON_AddNumberToObject(aff, "dur", paf->duration);
    cJSON_AddNumberToObject(aff, "mod", paf->modifier);
    cJSON_AddNumberToObject(aff, "loc", paf->location);
    cJSON_AddNumberToObject(aff, "bitv", paf->bitvector);
    cJSON_AddItemToArray(item, off); // add to the "Affc" array
  }
  if (item != nullptr)
    cJSON_AddItemToObject(o, "Affc", item); // add whole array of maps to o

  ...

  return o;
}

And next is the corresponding read section. This is a little more involved because of how ROM uses a memory pool for strings. We also use a variety of integer lengths and some strings that convert to bitpacked integers, so I defined some functions to handle the conversions:

#define STRKEY( literal, field, value )  \
  if ( !str_cmp( key, literal ) ) {      \
    free_string(field);                  \
    field = str_dup(value);              \
    fMatch = TRUE;                       \
    break;                               \
  }

#define INTKEY( literal, field, value )  \
  if ( !str_cmp( key, literal ) ) {      \
    field  = value;                      \
    fMatch = TRUE;                       \
    break;                               \
  }

void get_JSON_boolean(cJSON *obj, bool *target, const char *key) {
  cJSON *val = cJSON_GetObjectItem(obj, key);
  if (val) *target = (val->valueint != 0);
}

void get_JSON_short(cJSON *obj, sh_int *target, const char *key) {
  cJSON *val = cJSON_GetObjectItem(obj, key);
  if (val) *target = val->valueint;
}

void get_JSON_int(cJSON *obj, int *target, const char *key) {
  cJSON *val = cJSON_GetObjectItem(obj, key);
  if (val) *target = val->valueint;
}

void get_JSON_long(cJSON *obj, long *target, const char *key) {
  cJSON *val = cJSON_GetObjectItem(obj, key);
  if (val) *target = val->valueint;
}

void get_JSON_flags(cJSON *obj, long *target, const char *key) {
  cJSON *val = cJSON_GetObjectItem(obj, key);
  if (val) *target = read_flags(val->valuestring);
}

With that out of the way, reading data isn’t much different from the original Merc 2 way:

void fread_char(CHAR_DATA *ch, cJSON *json) {
  // need a field initialized first?  you can search for it:
  get_JSON_flags(json, &ch->act, "Act");

  for (cJSON *o = json->child; o; o = o->next) {
    char *key = o->string;
    bool fMatch = FALSE;

    switch (toupper(key[0])) {
    case 'A':
      // using the convenience KEY macros
      INTKEY("Alig", ch->alignment, o->valueint);

      // or a simple comparison to the key for complex types
      if (!str_cmp(key, "Atrib")) {
        get_JSON_short(o, &ch->stat[STAT_STR], "str");
        get_JSON_short(o, &ch->stat[STAT_INT], "int");
        get_JSON_short(o, &ch->stat[STAT_WIS], "wis");
        get_JSON_short(o, &ch->stat[STAT_DEX], "dex");
        get_JSON_short(o, &ch->stat[STAT_CON], "con");
        get_JSON_short(o, &ch->stat[STAT_CHR], "chr");
        fMatch = TRUE; break;
      }

      // or instead of using cJSON_GetObjectItem search, iterate
      // over the child objects
      if (!str_cmp(key, "Affc")) {
        for (cJSON *item = o->child; item != NULL; item = item->next) {
          int sn = skill_lookup(cJSON_GetObjectItem(item, "name")->valuestring);

          Affect *paf = new_affect();
          paf->type = sn;
          get_JSON_short(item, &paf->where, "where");
          get_JSON_short(item, &paf->level, "level");
          get_JSON_short(item, &paf->duration, "dur");
          get_JSON_short(item, &paf->modifier, "mod");
          get_JSON_short(item, &paf->location, "loc");
          get_JSON_int(item, &paf->bitvector, "bitv");

          paf->next       = ch->affected;
          ch->affected    = paf;
        }
        fMatch = TRUE; break;
      }
    }
  }
}

Wasn’t this about nested objects?

The original problem was that I was having trouble storing different types of nested lists in the Merc format. This turns out to be trivial with JSON, because the structure is already similar to the inventory hierarchy of the player. An object can contain other objects, or other named lists (“gems”, etc) as necessary. We simply create those lists within the cJSON object representing the game object.

cJSON *fwrite_objects(Character *ch, Object *head) {
  cJSON *array = cJSON_CreateArray();

  for (Object *obj = head; obj; obj = obj->next_content)
    cJSON_InsertItemInArray(array, 0, fwrite_obj(ch, obj));

  return array;
}

What about that spiffy recursion to write the list in reverse order? Turns out we don’t even need it – since cJSON stores children of a cJSON node as a linked list, we can efficiently insert new children at the head (index 0). This way, the list is written backwards, so we can naturally load it forwards. Next up is writing the object itself, which can write its own lists of contents as necessary:

cJSON *fwrite_obj(Character *ch, Object *obj) {
  cJSON *item;
  cJSON *o = cJSON_CreateObject(); // the object to return

  cJSON_AddNumberToObject(o, "Vnum", obj->pIndexData->vnum);

  // lots of if-checks here, no need to store data that the same as the prototype
  if (obj->name != obj->pIndexData->name)
    cJSON_AddStringToObject(o, "Name", obj->name);
  if (obj->condition != obj->pIndexData->condition)
    cJSON_AddNumberToObject(o, "Cond", obj->condition);
  if (obj->cost != obj->pIndexData->cost)
    cJSON_AddNumberToObject(o, "Cost", obj->cost);

  if (obj->wear_loc != WEAR_NONE)
    cJSON_AddNumberToObject(o, "Wear", obj->wear_loc);

  // you can add a whole array at once
  if (obj->value[0] != obj->pIndexData->value[0]
   || obj->value[1] != obj->pIndexData->value[1]
   || obj->value[2] != obj->pIndexData->value[2]
   || obj->value[3] != obj->pIndexData->value[3]
   || obj->value[4] != obj->pIndexData->value[4])
    cJSON_AddItemToObject(o, "Val", cJSON_CreateIntArray(obj->value, 5));

  // write the contents, if they exist
  cJSON_AddItemToObject(o, "contains", fwrite_objects(ch, obj->contains));

  // and other lists, if they exist
  cJSON_AddItemToObject(o, "gems", fwrite_objects(ch, obj->gems));

  return o;
}

We’re in the home stretch! All that’s left to implement is reading those objects back into the game. Here things get a little weird – we do a check to see if the object loaded correctly. What if we decided to remove the object from the game since last time the player logged on? Rather than blow up the contents, we put them into the player’s inventory.

void fread_objects(Character *ch, cJSON *contains) {
  if (contains == NULL)
    return;

  for (cJSON *item = contains->child; item; item = item->next) {
    Object *content = fread_obj(item);

    if (content->pIndexData) {
      obj_to_char(obj, ch);
    }
    else {
      // deal with object that has been removed from the game
      while (content->contains) {
        Object *c = content->contains;
        content->contains = c->next_content;
        obj_to_char(obj, ch); // don't just dump it on the ground
      }

      free_obj(content);
    }
  }
}

And finally, the code to load one game object:

Object * fread_obj(cJSON *json) {
  Object *obj = NULL;
  cJSON *o;

  if ((o = cJSON_GetObjectItem(json, "Vnum")) != NULL) {
    OBJ_INDEX_DATA *index = get_obj_index(o->valueint);

    if (index != NULL)
      obj = create_object(index, -1);
  }

  // if no object found, dummy one up so we can finish loading contents
  if (obj == NULL) { /* either not found or old style */
    obj = new_obj();
  }

  for (cJSON *o = json->child; o; o = o->next) {
    char *key = o->string;
    bool fMatch = FALSE;

    switch (toupper(key[0])) {
    case 'C':
      if (!str_cmp(key, "contains")) {
	// this mirrors code for fread_objects, but uses obj_to_obj
        // instead of obj_to_char.
        for (cJSON *item = o->child; item; item = item->next) {
          Object *content = fread_obj(item);

          if (content->pIndexData) {
            obj_to_obj(content, obj);
          }
          else {
            // ..., same as fread_objects
          }
        }
        fMatch = TRUE; break;
      }

      INTKEY("Cond", obj->condition, o->valueint);
      INTKEY("Cost", obj->cost, o->valueint);
      break;
    case 'N':
      STRKEY("Name", obj->name, o->valuestring);
      break;
    case 'W':
      INTKEY("Wear", obj->wear_loc, o->valueint);
      break;
    default:
      break;
    }
  }

  return obj;
}

What’s the downside?

“Wow!” you might say.  “Why not put all the game’s data in JSON?”

Why not?  JSON is easy to read, easy to write.  Players are probably the most complex thing you would ever use them for in a MUD.  Databases have their purposes (we use sqlite3 for some data) but files are good for human-accessible things.

I would say that the one thing I would not consider storing in JSON is the area files, unless you are wholly committed to OLC.  JSON is human readable, human changeable… but actually writing complex data files by hand would be tedious at best, and error-prone at worst.

You also might think that performance is an issue, because a JSON file is more verbose than the Merc format.  In the early 1990s, this was probably true.  I may have been a little hard on the developers of Merc in the introduction – they faced resource constraints and performance challenges that only embedded systems programmers think about now.  But in 2017, the extra 20-30% bytes in a JSON player file is going to amount to zero measurable difference.

Putting it all together

By now, I’ll bet you’re just dying to crack open your ROM source code for a marathon coding session, chugging Mountain Dew and pretending it’s 1994 again, dreaming up the new features made possible by the slick JSON player files. I tried not to make it too exciting, but it’s exciting stuff after all.

But wait. How do you get those old player files into the new format?

You could implement the new system in two stages, since your game already reads the old files.  Create the writer, and then come up with some way to batch load and save all of your current files.  I did that once to build a MySQL database pfile implementation, and it worked fine.  The parsing, anyway – SQL storage for complex game objects is a terrible idea.  Or, you could take the conversion script I wrote in Python and spend a few minutes adapting it to your particular flavor of pfiles.  I’ve included it in the source linked below.

I hope you’ve enjoyed this article, and for those of you still running old Merc/ROM derivatives, maybe it will be helpful.  You can find a copy of Legacy’s save.c here, and if you’d like to see this and our other projects in action, please visit us at legacy.xenith.org:3000.