70% of the web can’t be accessed by search engines. This means that most people don’t even realize they aren’t getting all the information available. The ethical and moral questions this brings up are far bigger than this post but, this opens up a universe of possibilities for the advanced seo agent. First, let’s get passed the ethical questions by highlighting a fantastic TED on Filter Bubbles.
Watch the Ted talk by Eli Pariser below:
Besides the increasing difficulty of getting all the information available to you there is another challenge called the invisible web.
“The Deep Web (also called Deepnet, the invisible Web, DarkNet dark Web or the hidden Web) refers to World Wide Web content that is not part of the Surface Web, which is indexed by standard search engines”
Wikipedia
Why is There an “Invisible Web”
There are several reasons that an invisible web exists.
- Dynamically Generated Pages
- Robots being blocked
- and much more…
[tabs slidertype=”top tabs”] [tabcontainer] [tabtext]Humanities[/tabtext] [tabtext]U.S. Government[/tabtext] [tabtext]Health & Science[/tabtext] [tabtext]Mega-Portals[/tabtext] [/tabcontainer] [tabcontent] [tab]Voice of the Shuttle: Started in 1994, VoS is one of the oldest and largest humanities databases on the Web, offering an exclusive index of content for subject ranging from Anthropology to Technology of Writing[/tab] [tab]• University of Michigan Government Documents Center: You’ll find reports, statistics, and other documents from all levels of the U.S. government here. Databases offered include Arts, Health Sciences, Social Sciences, and International Studies. • USA.gov: A red carpet portal into the depths of the United States government’s many entities. Includes government jobs, an A to Z list of governmental agencies, and information on finding grants, loans, and financial assistance. [/tab] [tab]• PsycNET: Use this database from the American Psychological Association to find abstracts and entire journals on various psychological topics (fees apply to complete journal entries, abstracts are usually free). • Scirus: A search tool dedicated only to scientific information, this amazing search tool has indexed hundreds of millions of scientific, scholarly documents to aid searchers all over the world in their pursuit of knowledge. • Healthfinder: Professionally vetted information from over one thousand different health databases on the Web. A wide range of resources on a dizzying array of health topics is available here. • RXList: If you’re looking for reliable drug information, then this database for prescription drugs, medications, and pill identifications is for you. Offers an A to Z index, image collections, and a dictionary of terms. [/tab] [tab]Mega-Portals • The University of California, Riverside maintains InfoMine, an incredible resource that at last count included over 100,000 links and access to hundreds, if not thousands, of databases. • The Virtual Library is simple and easy to use, with annotated subject links. I especially appreciate the annotations because it helps rule out extraneous search time. [/tab] [/tabcontent] [/tabs]
Competitive Intelligence (CI) is the formalized process of monitoring the competitive environment which is different than industrial espionage. Corporate espionage is a form of espionage don fo money. The Society of Competitive Intelligence Professionals (SCIP) defines CI as:
“a continuous process involving legal and ethical collection of information,analysis that doesn’t avoid unwelcome conclusions and controlled dissemination of information analysis.”
This is completely different than the ad hoc nature of traditional market research. I believe the online space is moving so fast in the current context of rapidly evolving markets, that developing a real time -on the fly system is the only way to keep ahead of competition.Although there are several moving parts to competitive intelligence the main component can be broken down into 5 key areas:
- Design and Setup. This 1st phase of the competitive intelligence is arguably the most important as it involves breaking down the requirement s for intelligence within the organization. In the past, CI was considered so secretive and so strategic it was only shared with the highest senior management and key staff in an organization. The emerging thought is that it should be shared with all levels of an organization.
- Information Collection. After determining the requirements an effective CI program establishes methods for collecting and archiving key information.
- Analysis. What makes CI so valuable is the insights gained from examining various informational variables. A good system should collect and archive the information. I think a modern short term way of doing this is to use scrapers like Mozenda to due this data collection in real time.
- Dissemination. Highly tailored analysis is then presented to the decision makers throughout the organization based on their individual requirements
- Feedback/System reset. A study published by Best Practices LLC found that company wide support was critical to succeed.
