Digging in the details: A case study in network data mining

Network Data Mining builds network linkages (network models) between myriads of individual data items and utilizes special algorithms that aid visualization of 'emergent' patterns and trends in the linkage. It complements conventional and statistically based data mining methods. Statistical approaches typically flag, alert or alarm instances or events that could represent anomalous behavior or irregularities because of a match with pre-defined patterns or rules. They serve as 'exception detection' methods where the rules or definitions of what might constitute an exception are able to be known and specified ahead of time. Many problems are suited to this approach. Many problems however, especially those of a more complex nature, are not well suited. The rules or definitions simply cannot be specified; there are no known suspicious transactions. This paper presents a human-centered network data mining methodology. A case study from the area of security illustrates the application of the methodology and corresponding data mining techniques. The paper argues that for many problems, a 'discovery' phase in the investigative process based on visualization and human cognition is a logical precedent to, and complement of, more automated 'exception detection' phases.