Json.NET by James Newton-King is the most popular .NET library for parsing JSON. The common use case is to read JSON data into a string and deserialize the text into an object, as in this example from Newton-King's website:
string json = @"{
'Name': 'Bad Boys',
'ReleaseDate': '1995-4-7T00:00:00',
'Genres': [
'Action',
'Comedy'
]
}";
Movie m = JsonConvert.DeserializeObject<Movie>(json); string name = m.Name; // Bad Boys
This technique works well for many situations. However, what happens when the string is gigabytes in size or is coming from a streaming source?
Very large data scenarios are becoming increasingly common and size does not need to reach terabytes before problems arise. Strings of even a gigabyte in length will cause significant problems. While I was stepping through code for a 64-bit app, I recently discovered a limitation in Visual Studio that causes the debugger to abort unexpectedly when it encounters strings about a gigabyte in size. Deserializing a gigabyte of JSON through DeserializeObject is simply not a well-supported use case.
To deal with these situations, Json.NET provides an excellent facility to tokenize streaming JSON. In my case, I was dealing with very large GeoJSON files containing GIS information. Many municipalities store very large amounts of GIS data in ESRI shapefiles that can be easily converted to GeoJSON using utilities (such as ogr2ogr) available with the Geospatial Data Abstraction Library. The data in these files include property boundaries, building perimeters, roads, and water features. For medium to large cities, the amount of data can easily be gigabytes in size.
Using Json.NET, I created a JsonTextReader object that reads such files, returning tokens one at a time. Even extremely large files can be read in seconds.
Basic Approach
The basic approach is to use a JsonTextReader object, which is part of the Json.NET library. A JsonTextReader reads a JSON file one token at a time. It, therefore, avoids the overhead of reading the entire file into a string. As tokens are read from the file, objects are created and pushed onto and off of a stack. When the end of the file is reached, the top of the stack contains one object — the top of a very big tree of objects corresponding to the objects in the original JSON file. The basic approach is shown in Listing One:
Listing One
Stack<JsonRoot> jsonStack = new Stack<JsonRoot>(); // declared at the class level
private void study1ToolStripMenuItem_Click(object sender, EventArgs e)
{
StreamReader streamReader = new StreamReader(@"bigdata.json");
JsonTextReader reader = new JsonTextReader(streamReader);
jsonStack.Clear();
while (reader.Read())
{
HandleToken(reader.TokenType, reader.Value);
}
}
The first line of code creates a stack of JsonRoot objects. The stack is a member variable of an enclosing .NET forms class. The method study1ToolStripMenuItem_Click is called in response to the click of a menu item called "Study 1." This method uses a hardcoded file name, "bigdata.json." In a real application, there will be additional code to prompt a user to select a file. However, for simplicity, I have left out those details. Note that the jsonStack is accessible to this method.
The first line of code of the method creates a standard .NET streamReader. The JsonTextReader constructor takes this streamReader as an argument and uses it for accessing the bytes in the file. The while loop iterates over the reader to retrieve the next token, which is stored in reader.Value as a .NET object. Reader.TokenType describes the object's type.
HandleToken is a method that uses the jsonStack to build a hierarchy of objects based on the tokens returned by the reader. Clear is called on the stack to ensure that it starts empty.
