Json.NET by James Newton-King is the most popular .NET library for parsing JSON. The common use case is to read JSON data into a string and deserialize the text into an object, as in this example from Newton-King's website:
string json = @"{ 'Name': 'Bad Boys', 'ReleaseDate': '1995-4-7T00:00:00', 'Genres': [ 'Action', 'Comedy' ] }";
Movie m = JsonConvert.DeserializeObject<Movie>(json); string name = m.Name; // Bad Boys
This technique works well for many situations. However, what happens when the string is gigabytes in size or is coming from a streaming source?
Very large data scenarios are becoming increasingly common and size does not need to reach terabytes before problems arise. Strings of even a gigabyte in length will cause significant problems. While I was stepping through code for a 64-bit app, I recently discovered a limitation in Visual Studio that causes the debugger to abort unexpectedly when it encounters strings about a gigabyte in size. Deserializing a gigabyte of JSON through DeserializeObject
is simply not a well-supported use case.
To deal with these situations, Json.NET provides an excellent facility to tokenize streaming JSON. In my case, I was dealing with very large GeoJSON files containing GIS information. Many municipalities store very large amounts of GIS data in ESRI shapefiles that can be easily converted to GeoJSON using utilities (such as ogr2ogr) available with the Geospatial Data Abstraction Library. The data in these files include property boundaries, building perimeters, roads, and water features. For medium to large cities, the amount of data can easily be gigabytes in size.
Using Json.NET, I created a JsonTextReader
object that reads such files, returning tokens one at a time. Even extremely large files can be read in seconds.
Basic Approach
The basic approach is to use a JsonTextReader
object, which is part of the Json.NET library. A JsonTextReader
reads a JSON file one token at a time. It, therefore, avoids the overhead of reading the entire file into a string. As tokens are read from the file, objects are created and pushed onto and off of a stack. When the end of the file is reached, the top of the stack contains one object the top of a very big tree of objects corresponding to the objects in the original JSON file. The basic approach is shown in Listing One:
Listing One
Stack<JsonRoot> jsonStack = new Stack<JsonRoot>(); // declared at the class level private void study1ToolStripMenuItem_Click(object sender, EventArgs e) { StreamReader streamReader = new StreamReader(@"bigdata.json"); JsonTextReader reader = new JsonTextReader(streamReader); jsonStack.Clear(); while (reader.Read()) { HandleToken(reader.TokenType, reader.Value); } }
The first line of code creates a stack of JsonRoot
objects. The stack is a member variable of an enclosing .NET forms
class. The method study1ToolStripMenuItem_Click
is called in response to the click of a menu item called "Study 1." This method uses a hardcoded file name, "bigdata.json." In a real application, there will be additional code to prompt a user to select a file. However, for simplicity, I have left out those details. Note that the jsonStack
is accessible to this method.
The first line of code of the method creates a standard .NET streamReader
. The JsonTextReader
constructor takes this streamReader
as an argument and uses it for accessing the bytes in the file. The while
loop iterates over the reader to retrieve the next token, which is stored in reader.Value
as a .NET object. Reader.TokenType
describes the object's type.
HandleToken
is a method that uses the jsonStack
to build a hierarchy of objects based on the tokens returned by the reader. Clear
is called on the stack to ensure that it starts empty.