Entropy Coding
H.264 defines two entropy coding modes, Context Adaptive Variable Length Coding (CAVLC) and Context Adaptive Binary Arithmetic Coding (CABAC).
CAVLC can be considered the "baseline" VLC. It is a conventional variable-length coding algorithm, with a table of uniquely-prefixed, variable-bit-length codes, but for additional efficiency the standard specifies additional tables. The selection among these tables and the length of the fixed-length coefficient value suffix is based on the local statistics of the current stream, termed the context.
CAVLC employs 12 additional code tables: 6 for characterizing the content of the transform block as a whole, 4 for indicating the number of coefficients, 1 for indicating the overall magnitude of a quantized coefficient value, and 1 for representing consecutive runs of zero-valued quantized coefficients. Given the execution efficiency of VLC tables, combined with this limited adaptive coding to boost coding efficiency, this provides a nice tradeoff between speed of execution and performance.
The CABAC mode has been shown to increase compression efficiency by roughly 10 percent relative to the CAVLC mode, although CABAC is significantly more computationally complex. In a first step, a suitable model is chosen according to a set of past observations of relevant syntax elements; this is called context modeling. If a given symbol is non-binary valued, it will be mapped onto a sequence of binary decisions, so-called bins, in a second step This binarization is done according to a specified binary, using a tree structure similar to a VLC code. Then each bin is encoded with an adaptive binary arithmetic coding engine using probability estimates which depend on the specific context. This pipeline is show in Figure 6.
Deblocking Filter
The last stage before reconstruction is a deblocking filter. This filter is intended to smooth the visual discontinuities between transform blocks, and as such is only applied to those pixels nearest these boundaries-at most 4 on either side of a block boundary. The filter consists of separable horizontal and vertical filters. Figure 7 shows the boundaries in a macroblock and the pixels of interest for a horizontal filter across a vertical boundary.
H.264 specified that the filter be applied on frames after de-quantization and before the image is used as a reference for motion compensation. For intra frames it should be applied after intra prediction.
This filtering is a very computationally expensive portion of the decoder, taking 15 to 30 percent of the CPU for low-bitrate streams that require the most filtering.
The deblocking filter is an adaptive filter, the strength of which is automatically adjusted according to the boundary strength and differences between pixel values at the border. The boundary strength is higher for intra blocks than inter, higher when the blocks in question have difference reference images, and higher when across a macroblock boundary. The pixel value differences must be less than a threshold that decreases with increasing quality. When the quantization parameter is small, increasing the fidelity of the compressed data, any significant difference is assumed to be an image feature rather than an error, so the strength of the filter is reduced. When the quantization step size is very small, the filter is shut off entirely. The encoder can also disable the filter explicitly or adjust it in strength at the slice level.
