The current state-of-the-art (e.g., Cisco NetFlow) methods which count periodically sampled packets are slow, inaccurate, and memory-intensive. However, keeping a counter to measure the traffic sent by each of a million concurrent flows is too expensive (using SRAM) or slow (using DRAM). Our algorithm not only dramatically improves the accuracy of flow distribution measurement, but also contributes to the field of data streaming by formalizing an existing methodology and applying it to the context of estimating the flow-distribution.ДЄccurate network traffic measurement is required for accounting, bandwidth provisioning, and detecting DOS attacks. Evaluations of this algorithm on large Internet traces obtained from several sources (including a tier-1 ISP) demonstrate that it has very high measurement accuracy (within 2%). Our algorithm uses Bayesian statistical methods such as Expectation Maximization to infer the most likely flow size distribution that results in the observed counter values after collision. The data structure is lossy in the sense that sizes of multiple flows may collide into the same counter. For each incoming packet, our algorithm only needs to increment one underlying counter, making the algorithm fast enough even for 40 Gbps (OC-768) links. In this paper we present a novel data streaming algorithm to provide much more accurate estimates of flow distribution, using a 'lossy data structure' which consists of an array of counters fitted well into SRAM. Its accuracy is limited by the (typically) low sampling rate required to make the sampling operation affordable. Previous work on estimating the flow size distribution has been focused on making inferences from sampled network traffic.

Knowing the distribution of the sizes of traffic flows passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and accommodate new traffic demands through better traffic engineering.

Active timer on netflow