Channels ▼


Looking For The Lost Packets: Part 2

Performance Problems

Performance problems become an issue when the output packets are correct and the system has been proven stable, but the performance metrics (like the throughput rate or the latency) do not meet the expected targets for some of the supported traffic scenario.

As the system is stable, it is arguably preferable to tackle performance tweaking problems than debugging stability problems. At least the performance problems are usually straightforward to reproduce. Some people would state that, once you have reached this stage, you can relax a little bit.

Possible Root Causes

If the performance problem is massive, then it might hide a functional or stability related problem. It may also be a design problem.

Design with the targeted performance numbers in mind from design day 1!

Otherwise, the performance tweaking problem might be caused by the incorrect dimensioning of resources when matched against the performance requirements.

Combat Techniques

Revisit the design, if need be.

Instrument the code with time stamps and profile the application to identify the problem regions that are candidates for optimization. Do not optimize those blocks that do not have a major contribution to the overall performance problem of the system. In theory, the blocks to pick for optimization should be those 20% of the blocks that use up to 80% of the packet budget.

Analyze the data collected by the real-time monitor looking for performance bottlenecks.

Overlooking some of the platform-related architectural considerations might also have a negative impact on the system performance:

  • Cache line: On Intel Architecture processors, the size of the cache line is 64 bytes. Try to maximize the number of cache hits by minimizing the number of cache lines that have to be used for storing your data structures. For example, if one data structure is less or equal to 64 bytes in size, by allocating it in memory on 64-byte aligned addresses, it will fit into a single cache line rather than spanning across two cache line. If two cache lines are used to store a structure which normally fits into a single cache, then the cache hit probability for accessing an instance of this data structure is cut in half.
  • Zero copy: Make sure that the packet does not have to be copied from one buffer to another in your design, as the memory copy is expensive.


In this article, I've presented several techniques for debugging functional, stability, and performance problems related to packet-processing systems.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.