Today I want to revisit the Invisible Web, and see if it is still relevant in 2026.
Welcome to Lead Prompt // executing leadership from the root. I’m your host, John Collins.
The Invisible Web is that part of the Internet that has somehow not been indexed by regular search engines. In the past is was widely predicted that the growth of the Invisible Web would outpace that of the public Internet, even before the advent of walled gardens like social networks.
Back in September of 2001, I made my own such predictions in the following blog entry on “The Invisible Web” that I will read to you now:
“Introduction
The amount of content on the Internet has grown exponentially within the past few years. Goldmines of information exist on the Net, if only you could find them! The World's leading search engine, Google.com, currently provides links to over 1.6 billion web pages. Even this staggering amount of data is nothing compared to the amount of content that is not indexed by the leading search engines, often referred to as the 'Invisible Web'.
Much of this information is in the form of publicly available databases, such as phone directories, newspaper archives or medical dictionaries. Due to the specialist nature of some of these databases, the content contained within them tends to be exactly what you are looking for and can be of a very high standard.
Search Engine Blues
Much of the difficulty of not being able to access the Invisible Web is the fault of the major search engines. Most major search engines employ automated programs called 'Spider Bots' or 'Crawlers' that literally 'crawl' their way through the Internet from link to link, indexing web pages as they hit on them. Unfortunately, if no links are found to a particular web site, the site will fall through the cracks into the Invisible Web, beyond the reach of the general search users who miss out.
Furthermore, even if a web site administrator submits his/her site to a major search engine, this is no guarantee that their site will be indexed. In fact, many search engines take several attempts to register a site, and on average take two to four weeks to process a submitted site. In the face of such problems, many sites go unchecked by the main search engines and fall through the cracks.
Search the Invisible Web
InvisibleWeb.com is a good place to start your search for more specialized information. It call's itself "the search engine of all search engines", and so long as you know roughly what you are looking for, it will provide you with a list of relevant sites to continue your search in more detail. For example, if you are looking for newspaper reports on famous incidents then you will only be provided with links for recognized newspaper archives, and not unrelated links that general search engines are so fond of throwing up at you.
The reality is that it is quite easy for anyone to set up a site and keep it quite covert in terms of the public's awareness of it. So long as the search engines cannot find it through links, and it is never submitted by anyone to the engines, it will effectively remain outside the public domain.
Considering that it is estimated that the Invisible Web is growing at a much faster rate than the rest of the Internet, it is possible to imagine a situation in the near future where the major search engines will simply not be able to cope with the pace of Internet expansion. Lets hope that they never become complacent about their technology, I for one will stick with Google as my default home page.”
It is always fascinating to me to revisit these old articles that I wrote a long time ago, in this instance 25 years ago, to see what I was concerned with at the time, and what predictions I was making.
For example, do you agree with my 2001 prediction that “it is possible to imagine a situation in the near future where the major search engines will simply not be able to cope with the pace of Internet expansion”, given that we are now living in that near future?
The InvisibleWeb.com is long-since gone from the Internet, and Gemini gave me the following summary when I asked about what happened to that project:
“The site’s path to disappearance involved several corporate hands:
- IntelliSeek (the owner) focused heavily on its metasearch engine, ProFusion, which also utilized "Invisible Web" technology.
- In 2006, IntelliSeek was acquired by Nielsen BuzzMetrics.
- Following the acquisition, the standalone InvisibleWeb.com domain was eventually shuttered or redirected as Nielsen pivoted the technology toward corporate data analytics and "social media monitoring" rather than consumer search.”
As of recording this episode, that domain is now dead.
The Internet remains undefeated.
Sources
The Invisible Web (2001) - https://leadprompt.sh/a/30-The-Invisible-Web
Download audio
File details: 7.6 MB MP3, 5 mins 43 secs duration.
Title music is "Apparent Solution" by Brendon Moeller, licensed via www.epidemicsound.com
Subscribe
Sponsor
Five.Today is a highly-secure personal productivity application designed to help you to manage your priorities more effectively, by focusing on your five most important tasks you need to achieve each day.
Our goal is to help you to keep track of all your tasks, notes and journals in one beautifully simple place, which is highly secure via end-to-end encryption. Visit the URL Five.Today to sign up for free!
