visualization, analysis, and next-gen tech

When you use traditional link analysis programs, you often are simply using a glorified etch-a-sketch. You add entities and connect them together with lines. And that’s where many of the commercial products end. Sure it makes it a little easier to visualize networks, but it is hardly an accurate or sophisticated methodology for what visualization and analysis should be able to accomplish.

Maybe you want to employ Social Network Analysis, and it certainly is a great tool for finding central players in a network. But both the etch-a-sketch and SNA routes aren’t going to yield accurate or comprehensive analysis unless the system takes metadata into account.

Let’s assume you are visualizing a terrorist network. You have your three key bad guys and the various people, places, and things associated with them. Then you have some connections directly between the three key bad guys. Use the etch-a-sketch approach and you could manually lay out the graph to support just about any kind of hypothesis. Add SNA and automatic layouts and now you are starting to see actual meaning. But what if SNA identifies Bad Guy A as the most central node in the network? And what if Bad Guy A is dead? Dead guys are hardly the most central node in a network after they are dead.

Mohammed Atta is the most important node
in the network according to SNA closeness…

…except that he is dead. Tell the algorithm about metadata
and you get a more accurate picture

Let’s look at another example. Take your same bad guy network, but add entities and relationships from different sources. For example, you have a cluster of entities based on information from reading classified intel. And another batch based on articles from the National Enquirer. The two sources are very different, the first would be deemed as highly credible, the latter as pure crap. Now run your SNA again. Remember that SNA is purely looking at nodes and edges. It will identify the most central players. But that centrality ignores the fact that sources affect the accuracy of the network data. Now throw in a few dead guys on top of your reliable and unreliable-sourced data and your centrality analysis is certainly suspect.

This disconnect between metadata and analysis is one of the key problems my company wrestled with when dreaming up the next generation in link analysis software, but more about that later.

A person’s status (dead or alive), and the credibility of the source that establish the entities and relationship are key metadata items that, when captured and analyzed, yield a huge increase in analysis accuracy. The type of the relationship (another key piece of metadata) also means something. To most etch-a-sketch programs, the relationship type is just a label that will be plastered on the screen. But to a system that understands metadata, the relationship type is a key weighting factor.

 Weighting….hmmm…

What if your system understood that a ‘Brother of’ relationship is stronger than a ‘Distant cousin of’? What if you could establish a weighting system that assigned relative weights to things like status (alive, dead, unknown) or reliability of source (high, medium, low) or familial relationships? And what if you could then temper the centrality measures provided by SNA with real metadata? The answer, a much more accurate model of any network: the power of SNA moderated by real-world ratings.

When I was working on the first version of the Sentinel link analysis product years ago, my team quickly identified reliance on metadata as a key way to enable accurate analysis. And so we built in rating values on all relationship types, on all status types, on credibility of information, reliability of source, and a host of other key metadata types. These values are used automatically in conjunction with traditional SNA to provide a degree of accuracy not previously available.

And we found out what we know today: metadata means something.