Skip to content
Home

Deep Web: definition, scope, history, and how it differs from the Dark Web

An explanatory overview of the Deep Web: what it is, why much content is not indexed, how it differs from darknets and the Dark Web, common uses and security implications.

Author: Created: Updated:

Overview

The Deep Web is the portion of the World Wide Web that is not indexed by ordinary search engines such as Google. Rather than being a separate network, the Deep Web consists of pages and resources living on the same infrastructure as the surface web but hidden from standard automated crawlers. Examples include content behind logins, private databases, dynamically generated pages, and paywalled material. Because these resources are not discoverable by crawling links alone, they do not appear in typical search results.

Key characteristics

Content commonly classified as Deep Web includes webmail, online banking portals, subscription academic journals, internal corporate sites, and pages created on demand by forms or scripts. Technical reasons for non-indexing vary: some sites block crawlers with protocols like robots.txt, others require authentication or session tokens, and many pages are generated dynamically via server-side queries that produce unique URLs only when requested. The Deep Web is generally believed to be substantially larger than the surface web, because so much of the Internet’s useful information is stored in databases rather than as static, linkable pages.

Common categories

  • Private or authenticated pages (e.g., webmail, dashboards)
  • Proprietary databases and archives (library catalogs, medical records)
  • Paywalled and subscription content (news or academic journals)
  • Dynamically generated pages and results behind forms

These categories are practical distinctions rather than separate networks. Many legitimate services rely on being non-indexed to protect privacy and commercial value.

Darknet and Dark Web: distinctions

A darknet is a private or overlay network that runs on top of the general Internet and requires special configuration or software to access. Examples of such overlays include networks that use anonymity tools. The term Dark Web describes websites and services hosted within darknets. Because darknets are inaccessible to conventional crawlers, the Dark Web is part of the Deep Web but is only a subset: all Dark Web content is non-indexed, but not all Deep Web content is part of a darknet.

Access to darknet-hosted services often involves dedicated tools and setup steps such as running Tor software, installing specialized software, or adjusting browser configuration to use nonstandard routing. These measures can obscure identifying information and make servers reachable only through the overlay network.

Origins and naming

The phrase "Deep Web" was introduced in the early 2000s by researcher Mike Bergman to highlight the vast amount of useful content not captured by standard search-engine indexing. The distinction helped draw attention to technical and legal issues surrounding discoverability, access, and preservation of online information.

There are legitimate reasons for keeping material off indexers: personal privacy, proprietary business systems, medical confidentiality, and controlled academic access. At the same time, parts of the Deep Web and especially services within darknets can be used to evade law enforcement or to distribute illegal goods and services. Activities such as piracy intersect with copyright law and copyright enforcement when protected works are shared without authorization.

Technical aspects of anonymity are relevant: ordinary Internet addressing exposes Internet Protocols (IP addresses) that can indicate a user’s approximate location or network. Darknets and anonymity tools try to reduce this visibility, but no method guarantees perfect secrecy. Researchers, journalists, law enforcement, and security professionals study the Deep Web and the Dark Web for both beneficial and forensic reasons.

Understanding the Deep Web requires recognizing it as a functional and technical category rather than a single place. It includes many everyday services essential to commerce, education, and communication while also encompassing spaces where privacy, censorship, and illicit activity intersect. For further reading on indexing, access methods, and legal frameworks, consult technical and legal sources on web architecture and internet governance.

Types of the Deep Web

According to Sherman & Price (2001), five types of the Invisible Web are distinguished: "Opaque Web", "Private Web", "Proprietary Web", "Invisible Web" and "Truly invisible Web".

Opaque Web

The opaque web are web pages that could be indexed, but are currently not indexed for reasons of technical performance or cost-benefit ratio (search depth, visit frequency).

Search engines do not consider all directory levels and subpages of a website. When capturing web pages, web crawlers steer through links to the following web pages. Web crawlers themselves cannot navigate, even get lost in deep directory structures, fail to capture pages, and cannot find their way back to the home page. For this reason, search engines often consider five or six directory levels at most. Extensive and thus relevant documents can be located in deeper hierarchy levels and cannot be found by search engines due to the limited indexing depth.

In addition, there are file formats that can only be partially captured (for example, PDF files, Google indexes only part of a PDF file and provides the content as HTML).

There is a dependency on the frequency of indexing a website (daily, monthly). In addition, constantly updated data sets, such as online measurement data, are affected. Websites without hyperlinks or navigation systems, unlinked websites, hermit URLs or orphan pages also fall under this category.

Private Web

The Private Web describes web pages that could be indexed but are not indexed due to access restrictions imposed by the webmaster.

These can be web pages in the intranet (internal web pages), but also password protected data (registration and possibly password and login), access only for certain IP addresses, protection against indexing by the Robots Exclusion Standard or protection against indexing by the meta tag values noindex, nofollow and noimageindex in the source code of the web page.

Proprietary Web

Proprietary Web refers to websites that could be indexed, but are only accessible after accepting a condition of use or by entering a password (free or paid).

Such websites are usually only accessible after identification (web-based specialist databases).

Invisible Web

The Invisible Web includes web pages that could be indexed from a purely technical point of view, but are not indexed for commercial or strategic reasons - such as databases with a web form.

Truly Invisible Web

Truly Invisible Web refers to web pages that cannot (yet) be indexed for technical reasons. These can be database formats that originated before the WWW (some hosts), documents that cannot be displayed directly in the browser, non-standard formats (for example Flash), as well as file formats that cannot be captured due to their complexity (graphic formats). In addition, there are compressed data or web pages that can only be served via a user navigation that uses graphics (image maps) or scripts (frames).

Databases

Dynamically created database web pages

Web crawlers almost exclusively process static database web pages and cannot reach many dynamic database web pages, as they can only reach deeper pages through hyperlinks. However, those dynamic pages can often only be reached by filling out an HTML form, which a crawler cannot do at the moment.

Cooperative database providers allow search engines to access the contents of their database through mechanisms such as JDBC, as opposed to (normal) non-cooperative databases that only provide database access through a search form.

Hosts and subject databases

Hosts are commercial information providers that bundle specialized databases of different information producers within one interface. Some database providers (hosts) or database producers themselves operate relational databases whose data cannot be retrieved without a special access facility (retrieval language, retrieval tool). Web crawlers do not understand the structure or language needed to read information from these databases. Many hosts have been operating as online services since the 1970s, and some of them operate database systems in their databases that predate the WWW.

Examples of databases: library catalogues (OPAC), stock exchange quotations, timetables, legal texts, job exchanges, news, patents, telephone directories, web shops, dictionaries.

Questions and answers

Q: What is the Deep Web?

A: The Deep Web is the part of the World Wide Web that cannot be searched on common search websites such as Google. It is also known as the Invisible Web or Hidden Web.

Q: Who first used the term "Deep Web"?

A: Mike Bergman, a computer scientist, was the first person to use the term "Deep Web" in 2000.

Q: Is darknet and Dark Web same thing as Deep Web?

A: No, they are not. A darknet is a type of computer network that is private or closed and it can be difficult to access. The Dark web is located in darknets and since no darknet can be found by Google or any other search website, it too falls under Deep web.

Q: What does IP stand for?

A: IP stands for Internet Protocol which contains important information about where a user is accessing the internet from.

Q: Why do people want privacy in internet?

A: People want privacy in internet for many reasons including doing things forbidden by governments such as piracy (sharing files protected by copyright laws).

Q: How do you access a darknet?

A: To access a darknet you would need to know a password, use specific computer programs, change your web browser configuration and other things depending on what network you are trying to access. Tor is an example of a commonly used darknet.

Q: What does 'piracy' mean?

A: Piracy means sharing files protected by copyright laws without permission from its owner/creator.

Related articles

Author

AlegsaOnline.com Deep Web: definition, scope, history, and how it differs from the Dark Web

URL: https://en.alegsaonline.com/art/26225

Share

Sources