If, when using a search engine, we went to the trouble and clicked on all the links displayed, we would see only a fragment of what is actually on the Internet. Google and other search engines are not all-knowing.
If someone wants to hide something on the Web from search engines, it will be easy to do. This state of affairs is used by, among others administrators of the so-called darknet. Darknet is conceptually similar to secret lodges in the real world. Initially, it had nothing to do with illegal machinations. Only a person who is personally known to at least one member of the lodge may attend the secret meeting. They take place in public places, but only the initiated know where and when.
Translated into the Internet language, this means that the darknets offer similar services to all other websites (e.g. websites, e-mail, file sharing), they are available to virtually everyone, provided that the interested party uses the right software and knows what and whom looking for. Darknets were created when law enforcement agencies around the world began to closely look at sites offering file-sharing and search for active participants in such portals.
File-swappers began looking for ways to keep them going and came up with a camouflaged counterpart to officially available peer-to-peer networks such as Napster, eDonkey and BitTorrent. In P2P networks, information exchange centers ensure that all users can download and upload MP3 files and movies without having to worry about copyright.
Exchange and chats with friends
The newly created networks, hidden within the Internet, were built on a P2P basis and renamed F2F (Friend to Friend). Unlike P2P networks, they do not have centralized servers, their participants cannot exchange files with anyone. Each user of the F2F network must know the IP address of their friends and have their digital business card (certificate), which is necessary to make contact.
This solution is to prevent external persons and law enforcement authorities from closing file-sharing services. One of the most famous F2F networks is Freenet. It uses the available Internet infrastructure and links for which we pay service providers, but isolates itself from the conventional Web. Freenet enables the exchange of files between users similar to the darknet and extends over many networks, within which there are classic websites (so-called freesites).
The intention of the creators of Freenet was to create a network on the Web, the users of which could remain anonymous, publish content without censorship and in which there would be freedom of expression. Citizens of countries such as China can freely exchange thoughts within Freenet, without fear of repression by the state. Secure communication takes place in private mode, the entire conversation is encrypted, and information is not transferred directly between interlocutors — the data is transmitted, just like in P2P networks, by the computers of other Freenet users. As a result, the official spy on the user’s actions will not be able to read the users’ correspondence.
Less private than exchanging messages in secret circles are freesites, websites distributed on Freenet that are not visible on the classic Web, and thus cannot be found by a search engine. Over time, Freenet has grown and grown a lot. An index of websites published on Freenet was also created, called Linkageddon. You cannot search this list, only browse it by scrolling the page.
There is real variety in the website directory — apart from pornography, there are also websites offering pirated copies of programs, games, movies and music. In addition to websites providing, for example, information about the oppression of the Tibetans by the Chinese government, we can come across websites run by lunatics who present the Auschwitz lie with pseudoscientific precision. The creators of Freenet did not foresee that freesites would swarm with content that falls under the paragraph, such as, for example, child pornography. Apparently, the authors of such sites take advantage of the lack of censorship to carry out illegal activities in a sense of impunity.
Darknets without twists
Billy Hoffman and Matt Wood, employees of the research department of IT giant Hewlett Packard, believe that darknets do not always have to be used to support illegal activities. Two years earlier, they demonstrated a new type of darknet called Veiled (veiled). “We are convinced that darknets would become more common if there were no barriers to downloading, installing and configuring software,” said Billy Hoffmann.
If access to the darknet had a wider audience, according to Hoffman, networks of this type would quickly find legal applications.
During the conceptual work, researchers considered using Veiled to publish Wikileaks documents. Administratory Wikileaks, instead of publishing documents on conventional Web sites that may be closed under legal or political pressure, they could publish them on a decentralized web. As a result, any attempt to block the distribution of documents would be a fight against windmills. Documents circulating around the darknet are not stored on several small servers, their parts are distributed among the computers of all darknet users. Freenet also works on the same principle.
Researchers from HP want to simplify darknets and opt to use for this purpose web browsers that allow access to such networks. So Veiled works — without having to download and configure the program each time — on computers with Windows, Mac OS X, Linux and iPhones. While the web browser may seem less powerful at first glance than a powerful darknet program, Veiled handles all the important functions of the hidden web, including encoding all data transfers and chatting between web users.
Hoffmann and Wood shared extensive Veiled documentation, thereby motivating other developers to follow their concepts. No matter how clever Veiled is, its authors decided not to publish a ready-made program. The reasons are legal reservations on the part of the employer. The authors therefore posted a presentation on the Internet, the informative value of which allows other developers to easily create a Veiled clone.
Deep Web greater than the World Wide Web
The Deep Web is also invisible to search engines and their users, but quite differently organized than the darknets. According to experts’ estimates, it is over 1000 times larger than the traditional Internet. On the one hand, the Deep Web is a huge data bank, the content of which is not understandable to popular search engines. Only when the user sends a special query to the database will it return the desired result. On the other hand, the Deep Web owns servers that do not let search engine robots inside. They do it in two ways: either with a login or via special messages to a web crawler (a robot that searches the website).
Each website administrator can determine whether and which parts of his website may be included in the index of Google and other search engines. If the site administrator does not admit the web crawler, the web crawler will not appear in the search results list, even if 10,000 other cataloged web pages are linked to that site. Under normal circumstances, many connections guarantee a high position in the search results list. Google’s hawk eyesight fails in the described situation and the search engine shows users only a fragment of what is swirling on the Web.
Library catalogs are examples of the existence of the Deep Web. Although the lists of books and magazines are stored on conventional web servers, we will gain access to web resources after entering a login and password, and search engines do not have these. So they can only look at the closed door from the outside — they won’t get anything wrong. The same is true for the flight information dataset, medical professional advice, product data and much more.
It is true that Google solved the problem with flight information by purchasing ITA software (American Google users can, thanks to ITA, enter a search query for the most advantageous connection from New York to Las Vegas), but the search engine still does not reach into the depths of the Deep Web . Search engines such as BrightPlanet promise to help access the depths of the Deep Web, but only for a fee.
Getting started with the hidden network
Even without special software installed, we can immerse ourselves in the Deep Web — all you need is the help of appropriate portals and search engines.
oaister.worldcat.org A project initiated by the University of Michigan. OAlster (read: ojste) searches the metadata of a server containing documents from over 400 institutions around the world, including resources from libraries, research institutions, specialist journals and many more. They contain data that is of interest primarily to scientists. OAlster offers access to approximately 18 million databases.
scirus.com An extremely powerful research tool — the scientific search engine Scirus stores an index of 440 million entries: from photos, through magazine articles, to websites.
deepwebresearch.info Mark Zillman has accumulated a collection of links to articles, documents, message boards and videos on the Deep Web. The catalog is a good introduction to the subject of the hidden Web, although it is available in a rather unclear form of a list saved as a PDF document.
louvre.fr The museum’s website is no leap into the deep Web, but if we choose “Search the Collection” and “Database” — six databases will open describing the museum’s collections that we cannot find using Google.
Onion Routing
By using an ordinary web browser, we won’t get into the darknet. For this, we will need add-ons that allow access to the hidden network. The gateway supports the so-called onion routing, which is a complex anonymous access technique that passes data through a string of ever-changing encrypted proxies. This system is also used on the darknet.
It’s not hard to understand at this point, but it gets more complicated: there are no standard URLs in the darknet. Internet pages are accessed through coded combinations of letters and numbers, ending in “.onion”. These addresses change frequently and therefore must be found constantly — but we will not tell you how to find them.
Comments
Post a Comment