Channels ▼
RSS

.NET

Parsing Big Records with Json.NET


Json.NET by James Newton-King is the most popular .NET library for parsing JSON. The common use case is to read JSON data into a string and deserialize the text into an object, as in this example from Newton-King's website:

string json = @"{
  'Name': 'Bad Boys',
  'ReleaseDate': '1995-4-7T00:00:00',
  'Genres': [
    'Action',
    'Comedy'
  ]
}";
Movie m = JsonConvert.DeserializeObject<Movie>(json);

string name = m.Name;
// Bad Boys

This technique works well for many situations. However, what happens when the string is gigabytes in size or is coming from a streaming source?

Very large data scenarios are becoming increasingly common and size does not need to reach terabytes before problems arise. Strings of even a gigabyte in length will cause significant problems. While I was stepping through code for a 64-bit app, I recently discovered a limitation in Visual Studio that causes the debugger to abort unexpectedly when it encounters strings about a gigabyte in size. Deserializing a gigabyte of JSON through DeserializeObject is simply not a well-supported use case.

To deal with these situations, Json.NET provides an excellent facility to tokenize streaming JSON. In my case, I was dealing with very large GeoJSON files containing GIS information. Many municipalities store very large amounts of GIS data in ESRI shapefiles that can be easily converted to GeoJSON using utilities (such as ogr2ogr) available with the Geospatial Data Abstraction Library. The data in these files include property boundaries, building perimeters, roads, and water features. For medium to large cities, the amount of data can easily be gigabytes in size.

Using Json.NET, I created a JsonTextReader object that reads such files, returning tokens one at a time. Even extremely large files can be read in seconds.

Basic Approach

The basic approach is to use a JsonTextReader object, which is part of the Json.NET library. A JsonTextReader reads a JSON file one token at a time. It, therefore, avoids the overhead of reading the entire file into a string. As tokens are read from the file, objects are created and pushed onto and off of a stack. When the end of the file is reached, the top of the stack contains one object — the top of a very big tree of objects corresponding to the objects in the original JSON file.  The basic approach is shown in Listing One:

Listing One

Stack<JsonRoot> jsonStack = new Stack<JsonRoot>(); // declared at the class level

private void study1ToolStripMenuItem_Click(object sender, EventArgs e)
{
    StreamReader streamReader = new StreamReader(@"bigdata.json");
    JsonTextReader reader = new JsonTextReader(streamReader);
    jsonStack.Clear();

    while (reader.Read())
    {
        HandleToken(reader.TokenType, reader.Value);
    }
}

The first line of code creates a stack of JsonRoot objects. The stack is a member variable of an enclosing .NET forms class. The method study1ToolStripMenuItem_Click is called in response to the click of a menu item called "Study 1." This method uses a hardcoded file name, "bigdata.json." In a real application, there will be additional code to prompt a user to select a file. However, for simplicity, I have left out those details. Note that the jsonStack is accessible to this method.

The first line of code of the method creates a standard .NET streamReader. The JsonTextReader constructor takes this streamReader as an argument and uses it for accessing the bytes in the file. The while loop iterates over the reader to retrieve the next token, which is stored in reader.Value as a .NET object. Reader.TokenType describes the object's type.

HandleToken is a method that uses the jsonStack to build a hierarchy of objects based on the tokens returned by the reader. Clear is called on the stack to ensure that it starts empty.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Comments:

Video