Basic Knowledge Should Be Free

80% of SEO is the same for everyone… For the SEO industry to evolve, everyone must be armed with the same basic facts. The only way to accomplish that is to make the basic information available to all, for free.

Not just free, but well organized. The web is free, once you have access, and there’s not much in this post that you couldn’t find on the web… But the web is not well organized.

And let’s be honest, there’s no editorial filter on the web – lies look just as good as the truth when you have a good designer, and mistakes and misinformation doesn’t  come with warning labels.

The web, by itself, is not a solution.

For information to be really useful, it has to be collected, reviewed, edited, organized, and written by someone who knows their stuff.

Someone has to do it.

I can afford to do it.

So here we are.

SEO Is Just the Beginning - Five More “Not-So-Easy” Pieces

I came onto the web with a background in advertising, marketing and sales.

I was an “SEO guy” as far as anyone was concerned. Now, I think I’ve done “OK” in SEO, but that’s not all I do.

How Search Engines Work

Before we get into the SEO process itself, let’s take a look at how search engines work – both from our perspective (crawling and indexing) as well as from the searcher’s side of things.

The actual technical details are much less important than you’ve probably been led to believe, so I’ll give you a quick walk-through of what search engine spiders do, instead of boring you to death with detail.

 In reality, the search engines are more alike than they are different.

While there are still significant differences between them in some areas, you can learn how to stay on the right side of those differences.

Why It Matters To You

There is a lot of very bad advice out there on search engine positioning. A lot of this bad advice is for sale, and this market has created a lot of self-reinforcing myths about the subject.

Many of these myths are really just self-serving propaganda created by various players in the search engine positioning industry, and repeated by well-meaning writers in e-business publications and other media outlets.

 Think about this for a moment. Where does “conventional wisdom” come from?

Well, it comes from various media outlets, such as television, radio, print, and the Internet itself. But where does a writer from a business magazine find out about any industry? They get their information from the “experts” in the industry itself.

Reporters don’t have a lot of time to do research, so a lot of what you see in print is just repackaged press release material and the “industry line.”

 In reality, following “conventional wisdom” without thinking can sometimes do you a lot of harm. A lot of people spend a lot of time trying to make their pet schemes sound like “standard practice.”

In the process of becoming search engine literate, you’ll learn that a lot of the things you’ve been told are just plain wrong. You’ll also discover that this whole subject is much less intricate and mysterious than many “experts” want you to believe.

My intent here is to share the knowledge I have, and let you make your own informed decisions. Why should you care about how search engines work? By investing the time it takes to read this, you’ll be better prepared to understand the rationale behind everything we do.

What a “Spider” Does

The first thing that you need to understand is what a search engine “spider” is, and how it works. A "spider" (also known as a "robot" or "crawler") is a software program that search engines use to find what’s out there on the ever-changing web.

 There are many types of spider in use, but for now, we’re only interested in the one that actually “crawls” the web finding pages. This is a somewhat oversimplified picture, but basically this program starts at a website, loads the pages, and follows the hyperlinks on each page.

In this way, the theory goes, everything on the web will eventually be found, as the spider crawls from one website to another. Search engines may run thousands of instances of their web-crawling spider programs simultaneously, on multiple servers.

 When a "crawler" visits one of your web pages, it loads the page’s contents into a database. Once a page has been fetched, the text of your page is loaded into the search engine’s index, which is a massive database of words, and where they occur on different web pages.

So there are really three steps. It starts with crawling (fetching pages), then indexing (breaking them down into words for the index), and a final step where the links (web page addresses / URLs) that are found get fed back into the crawling program to be retrieved.

 When the spider (some of them will check later to verify that a page really is offline) doesn't find a page, it will eventually be deleted from the index. This is one reason why it’s important to use a reliable web hosting provider.

 The first thing a spider is supposed to do when it visits your site is look for a file called “robots.txt”. This file contains instructions for the spider on which parts of the web site to index, and which to ignore. The only way to control what a search engine spider sees on your site is by using a robots.txt file.

 All spiders are supposed to follow certain rules, and the major search engines do follow these rules for the most part. We live in exciting times, though, and the major search engines are finally working together on standards. The latest standard is the XML Site Maps protocol.

How They Find You

The most common way that search engine spiders find a website is by following hyperlinks from other sites. In search engine terminology, these are known as “found pages.”

Some search engines also have a “submit URL” form, where you can request that they add your web site to their index. Typically, you just give the primary URL for your site (like http://www.IndioCertifiedSmog.com), and this address is added to their list of links to crawl.

 There are services (some paid, some free) that let you submit your site “automatically.” This is not really a good deal, even if it’s free. At this point, there are only four “major” search engines, and you can’t even submit to all of them.

 The same applies to software that submits your site to “thousands” of search engines. As I’ve just stated, only a few search engines actually matter. Using this software is more likely to generate a flood of junk email than anything else.

 If all of this sounds like I’m recommending you let the search engines find you, rather than submitting your site, I am.
I have not submitted anything to the search engines in years, and it rarely takes more than a week or so for them to find and crawl my new sites.

Focus on getting links set up – this will not only improve your rankings, it will ensure that the search engines can find you on their own.

 Some search engines (like Yahoo) offer a “paid inclusion” program. Although you get no special advantage from using it, in terms of your site’s ranking, it does ensure that your site’s indexing is up to date. This isn’t necessarily a great deal, since it usually involves paying by the click for traffic you may have gotten anyway.

Stay Tuned Up,
Dale