Nasa joins US government project to create 'Google for the dark net' that could uncover cyber criminals, paedophiles and drug dealers in the online underworld

By Mark Prigg For Dailymail.com

19:31 25 May 2015, updated 20:55 25 May 2015

The deep web is a hive of illegal activity, rife with child pornography, drug deals and human trafficking.

But because it is 'buried' so deep it is considered out of the reach of mainstream search engines and many law enforcement agencies - until now.

Nasa has revealed it is joining a Defense Advanced Research Projects Agency (Darpa) project to scour content on this so-called dark net.

Called Memex, it is a combination of memory and index - that not only indexes, but also identifies subtle patterns in activity.

'When you do a simple Web search on a topic, the results that pop up aren't the whole story,' said Nasa.

The Internet contains a vast trove of information - sometimes called the 'Deep Web' - that isn't indexed by search engines: information that would be useful for tracking criminals, terrorist activities, sex trafficking and the spread of diseases.

According to Nasa, scientists could also use it to search for images and data from spacecraft.

Memex could, for example, help catalog the vast amounts of data NASA spacecraft deliver on a daily basis.

'We're developing next-generation search technologies that understand people, places, things and the connections between them,' said Chris Mattmann, principal investigator for JPL's work on Memex.

Memex checks not just standard text-based content online but also images, videos, pop-up ads, forms, scripts and other ways information is stored to look at how they are interrelated.

'We're augmenting Web crawlers to behave like browsers - in other words, executing scripts and reading ads in ways that you would when you usually go online.

'This information is normally not catalogued by search engines,' Mattmann said.

Additionally, a standard Web search doesn't get much information from images and videos, but Memex can recognize what's in this content and pair it with searches on the same subjects.

The search tool could identify the same object across many frames of a video or even different videos.

The video and image search capabilities of Memex could one day benefit space missions that take photos, videos and other kinds of imaging data with instruments such as spectrometers.

Searching visual information about a particular planetary body could greatly facilitate the work of scientists in analyzing geological features.

Scientists analyzing imaging data from Earth-based missions that monitor phenomena such as snowfall and soil moisture could similarly benefit.

Memex would also enhance the search for published scientific data, so that scientists can be better aware of what has been released and analyzed on their topics.

The technology could be applied to large NASA data centers such as the Physical Oceanography Distributed Active Archive Center, which makes NASA's ocean and climate data accessible and meaningful.

Memex would make PDF documents more easily searchable and allow users to more easily arrive at the information they seek.

Awareness of existing publications also helps program managers to assess the impact of spacecraft data.

All of the code written for Memex is open-source.

JPL is one of 17 teams working on it as part of the DARPA initiative.

Memex is related to DARPA's previous Big Data initiative called XDATA, managed by DARPA Program Manager Wade Shen.

That research effort is also aimed at processing and analyzing large amounts of data, with defense, government and civilian applications. JPL was one of 24 groups involved.

'We are developing open source, free, mature products and then enhancing them using DARPA investment and easily transitioning them via our roles to the scientific community,' Mattmann said.

Darpa said: 'Today's web searches use a centralised, one-size-fits-all approach that searches the internet with the same set of tools for all queries.

'[But] common search practices miss information in the deep web - the parts of the web not indexed by standard commercial search engines - and ignore shared content across pages.'

Memex was designed to overcome these challenges by extending 'the reach of current search capabilities and quickly and thoroughly organise subsets of information based on individual interests.'

It looks behind standard search results for patterns, links and similar behaviours.

The software scours all aspects of the web - including those hidden in the dark net - to create data maps that might reveal clues about illegal activity.

In particular, Darpa wants to use Memex to uncover human trafficking rings by searching for patterns in the number of online sex adverts being posted from certain regions, or porn sites featuring the same email addresses or phone numbers.

MEMEX HISTORY

The Memex program gets its name from a hypothetical device described in 'As We May Think' - a 1945 article for The Atlantic Monthly.

It was written by Vannevar Bush, director of the US Office of Scientific Research and Development (OSRD) during World War II.

In the article, Memex was described as an analog computer that would supplement human memory.

It would store and automatically cross-reference all of the user's books, records and other information.

This cross-referencing, which Mr Bush called 'associative indexing', would let users quickly search large amounts of information, and gain insights from it.

These patterns could reveal links that human investigators could miss, explained Scientific American.

'We're envisioning a new paradigm for search that would tailor indexed content, search results and interface tools to individual users and specific subject areas, and not the other way around,' said Chris White, Darpa program manager.

'By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualise access to information. Ease of use for non-programmers is essential.'

Targeting the deep web is also an initiative being developed in the UK.

TRACKING CHILD ABUSE VIDEOS

Facebook, Microsoft, Google, Twitter and Yahoo will use proposed digital fingerprints, or hash values, of thousands of known child sex abuse images to prevent them being shared.

Google has also developed new technology that allows child abuse videos to be identified and blocked.

Microsoft, Google and Mozilla have also said they will investigate ways of placing restrictions within their browsers to block people from accessing child pornography.

Google and Microsoft have both also introduced a number of changes to their search functions globally.

Google claims to have seen a five-fold reduction in the number of searches for child abuse images since these changes were made.

However, these changes are thought to be partly responsible for driving much of the trade in child pornography into the dark net.

In December, the UK government said a specialist unit was being set up to hunt down paedophiles using the dark net to share child pornography.

The National Crime Agency and the Government Communications Headquarters (GCHQ) will use advances in analysing images and communications to trace the 'digital footprints' left by the users who share them.

Prime minister David Cameron said the new unit is aimed at 'shining a light on the web's darkest corners' as he announced a package of measures to tackle online child abuse.

The National Crime Agency estimates that around 20,000 people from the UK use secret or encrypted networks each day.

The dark net consists of a network of encoded websites that sit behind the publicly available websites and cannot be found with normal search engines.

It came to prominence in 2012 when the FBI made a series of raids on Silk Road - an online marketplace described as the 'eBay for illegal drugs'.

Figures compiled by the National Crime Agency suggest that use of the dark net rose by two thirds in 2012.

Hidden capabilities that let users email and host file storage through encrypted and anonymous networks are provided by services the The Onion Router (Tor).

Tor users currently represent 0.18 per cent of the total number of internet uses in the UK. ( percuma dong anon tapi data jumlah penggunanya di buka)

However, in a recent investigation, GCHQ and the NCA were able to track down a British man who had been maintaining chat rooms and websites in the Far East and Eastern Europe that were used to share child pornography around the world.

He was also offering advice to other paedophiles about how to hide their behaviour and was using software to keep himself anonymous.

Analysis by experts at GCHQ allowed them to trace the man and he was later sentenced to three years in prison for making and distributing indecent images of children.

Cameron & GCHQ to krack down on 'paedophiles and perverts'