Catching terrorists with spidering

In small rooms, down dark halls, computer scientists are doing their part to fight against terrorism. The battle to win against the terrorists can't be won just on the battlefield. The internet has proved to be a vital tool both for the terrorists and in the fight against the terrorists.

Working from a windowless room accessed after passing through a labyrinth of security systems at the University of Arizona, Hsinchun Chen typifies the new breed.

Four years ago, Chen, a computer scientist, created a project known as the Dark Web. As the name suggests, the idea was to access the dark corners of websites where radicals hide and attempt to recruit. Security and intelligence services were inadequate at tracking these sites, Chen says, and they were looking to private industry or academics for help.

Chen now boasts of having the world's largest collection of terrorism websites and is one of a handful of experts on contract to the U.S. government.

With a technique known as "spidering," the Dark Web continually crawls through the Internet finding the back doors into chat rooms and protected websites where some of today's terrorist plots are hatched.

As one aspect of his program, Chen developed a mathematical formula to follow radical discussion threads.

With the overwhelming volume of Web material, this algorithm helps pinpoint what Chen calls the "infectiousness" of violent ideas.

"In my perception, there's a small percentage in this Internet world that are very non-violent, mainstream and so on, peaceful and willing to state their opinion," says Chen.

"Then, there are the violent ones who also make up a minority. Ninety per cent are in the middle. These are people who can be flipped, especially the young people." source

Cross posted at AntiDhimmi