Michail Vlachos

pcapIndex: an index for network packet traces with legacy compatibility

Francesco Fusco, Xenofontas Dimitropoulos, Michail Vlachos, Luca Deri
Appears in: 
CCR January 2012

Long-term historical analysis of captured network traffic is a topic of great interest in network monitoring and network security. A critical requirement is the support for fast discovery of packets that satisfy certain criteria within large-scale packet repositories. This work presents the first indexing scheme for network packet traces based on compressed bitmap indexing principles. Our approach supports very fast insertion rates and results in compact index sizes. The proposed indexing methodology builds upon libpcap, the de-facto reference library for accessing packet-trace repositories.

Public Review By: 
Philip Levis

The pace of technological research can be dizzying, and the past decade has been especially amazing for networking. We've seen Internet traffic swing from the web to BitTorrent to Netflix; we’ve seen the source worms and hosts of other attacks expand from individuals to organized crime; we've seen whole new classes of networks, such as data centers, emerge. In a world where technology changes so quickly, stable, well-designed tools whose utility has stood the test of time are critically important. Tools let easily observe and understand networks even as they evolve. The longevity of tools such as tcpdump on one hand demonstrates their utility. On the other hand, it also shows their age: they were designed a long time ago. While networking has moved forward, so have many other fields, and applying new techniques to old problems can be a valuable contribution, deepening our understanding of networks. This paper is an excellent example of such a contribution. Tcpdump is a tool we've all used at some point. If you’ve ever tried to use tcpdump to search enormous packet traces, you know it can be tremendously slow. Franceso Fusco and his co-authors show how by changing tcpdump’s underlying library, libcap, to use state-of-the art bitmap indexes, one can in some cases speed it up by three orders of magnitude. In the worst case they observe, the speedup is a factor of 1.9. The reviewers for the paper agree that using bitmap indexes for packet traces is not a revolutionary new idea: it takes existing techniques from the database community and applies them to a problem in networking. But they also agree that the performance gains are impressive, and that the result is of significant practical benefit to networking researchers and engineers. The contribution of this paper lies in identifying an otherwise overlooked but very real problem and designing a good solution with cutting edge technology. Papers such as this one form the technical underpinnings of the tremendous gains we have seen in networks over the past decade and we hope to continue to see in the future.

Syndicate content