Search engines are by far the most popular tool used to find Web sites, yet most sites are designed with no knowledge of how search engines work. Many are essentially search engine-invisible.
There are two kinds of Web search services that are commonly called "search engines" --directories and true search engines. They are very different. Each has its advantages and drawbacks for searching. Both are important if you want people to find your site.
Search Engines and Directories
Directories are categorized listings of selected Web sites compiled by people. Human editors review sites and edit descriptions that Webmasters submit. Directories can be browsed or searched, but searches usually are limited to titles, descriptions, and category names as they appear in the directory. Important directories are Yahoo, Open Directory, and LookSmart.
Search engines, on the other hand, use software agents called "spiders" or "crawlers" to index the text content of the Web, page by page. The full content of the indexed pages is stored in a very large searchable database. Searches return Web pages, ranked in order of relevance, that are likely to match the entered search terms. Text content and frequency, page structure, and various off-page factors are all considered to determine rankings. AltaVista, HotBot/Inktomi, Excite, and Google are among the important search engines.
Increasingly, search engines and directories are evolving into search portal sites that combine both methods of searching. They're also incorporating search technologies intended to make searches more relevant and user-friendly, and to eliminate spam. As a result, search optimizing must take into account a growing number of factors.
Choosing Search Terms
To optimize a site so it can be found, you first need to determine what words people will use in searching. This is important both in optimizing for search engines and in submitting to directories.
People usually search in groups of words or phrases. It's extremely difficult to optimize for a single word search. Ask for suggestions from clients and associates; look at your competition; run searches yourself; and start making a list.
There are a number of resources for seeing how people actually search online. One of the most useful is the Search Term Suggestion List on GoTo.com. MSN's bCentral has a new beta keyword suggestion tool. Compare the list you've compiled with results from these, and boil it down to 20 or 30 prioritized words and phrases that might be used to find your site.
Text Frequency And Page Structure
Though the major search engines all differ, the basic on-page elements they consider are generally the same. Since they see HTML text strings only, engines can't really understand the sense of a page. They must interpret relevance by text frequency and page structure.
Most engines use a page model that has evolved from fundamental HTML document structure and its underlying logic. They assume that the title you assign to a page should be descriptive of its contents; that headings on a page are there for organization and emphasis and are important; and that the text at the top of the page is generally more important than the text lower down... all pretty basic.
They also assume that search relevance, in the various sections of the page, is proportional to word frequency; but they penalize for obvious attempts at manipulation.
Following from this model of page structure, some elements of a page carry more "weight" or importance for search than do others. Of the meta tags, the title is by far the most important. It becomes a core around which you can focus your optimization.
In the body text, H1 headings carry the greatest weight. The first 250 characters in your body copy also get special importance on some engines. Link text generally carries extra weight as well.
Focusing Your Pages
Page focus is a main key to search. As the page title is generally the strongest single optimization element you have, it's a good place to start. Build a title using targeted search word phrases, and have text content that relates to this title at the most effective places on the page. But don't try to cram a lot of search terms into any one page. They will dilute each other, lose relevance, and make the page impossible to read.
Consider a total site strategy. If you want to present a broad target, you must have various searchable pages, each focusing on different search terms people are likely to use.
Meta Titles
For search, titles should be less than 70 characters long, including spaces. Fifty or 60 are probably better. Thus, each title will accommodate only 4 to 6 words from your search term list.
Using your most important search terms first, try to come up with page titles that read well, are appropriate to page content, and will encompass several likely search phrases. "Automotive Engineering & Structural Materials Development," e.g., also incorporates "automotive materials," "materials engineering," etc.
Most default searches are all-the-words rather than exact phrases. If there is an exact phrase that you absolutely want to make sure is covered (eg, "structural engineering"), try to keep that phrase intact... or keep it intact later on your page and see if that is sufficient.
Use only essential words, and try not to repeat search words within the title. Try never to repeat words consecutively anywhere on your page.
Meta Description And Keywords
The meta description should try to include all of the search word phrases contained in your title. Keep in mind also that your meta description and title are what's displayed in most search engine results. Once your site has been located, a compelling description is important to attract the searcher to your site.
Meta keywords are widely misunderstood and misused. Putting a word in your keywords tag does not usually mean that you will rank well on that word. In fact, using too many meta keywords, as many Web designers do, can actually reduce the prominence of desired search words on your page.
It's best to think of keywords as providing emphasis for important search words that are already in your body text. Apart from an occasional synonym or misspelling that you may want to include, your meta keywords should contain only words that are on your page. Be careful of excessive word repetition.When in doubt, less is probably better.
You should have different meta tags for every searchable page on your site.
Body Text
For good search results, your meta tags and page content must work together. The title and meta tags tell the engine what your page is about. Whether this is confirmed by the body text has a lot to do with how effective those meta tags will be.
Remembering the importance assigned to H1 headings and to text at the top of your page, here's a basic, albeit very mechanical, procedure for optimization:
-
repeat your title tag in an H1 heading right at the top of the page. - use the phrases in your title twice near the top of your page (within the first 250 characters).
-
repeat the phrases further down the page, both in regular text and in subheadings.
-
include search words in image ALT tags.
A search word density of between 3-6% is in the ball park for good placement on most search engines.
H1 headings, though, can be ugly, and exact phrase repetitions throughout a page can be awkward. It is possible to construct a good page this way, but it's challenging. You don't have to follow this formula exactly.
What is important is to know the principles involved. You want to get the search words in your title echoed in your text, and have them at strategic spots on your page.
You might prefer simply to take a look at your existing pages, check the word frequencies, decide what words and phrases are most significant for each page, tweak the copy, and focus your meta tags and page titles around them. This can often help.
If you're really fighting for search position on competitive terms, though, you'll need all the relevancy boosters you can get. Even table structure on a page can affect search engine ranking. You may decide to create "doorway" pages, focused enough to rank well on important search terms, but attractive enough to get visitors to enter your site.
Spamming, Pitfalls, And Limitations
More is not always better. Early on, aggressive Web marketers prompted the search engines to install spam filters. Excessive text repetition, particularly consecutive repetition, either in body text or meta tags, can bring on spam penalties. "Sex sex sex" is not necessarily better than "sex," and it could be worse. Keyword stuffing in ALT tags will either be ignored or penalized. In the same vein, mirrored sites on multiple domains are often banned from search engines.
The trick of "hidden" text, where the font color is the same as the body background color, can get a domain banned from most search engines. This is something that can happen inadvertently, so designers need to beware. White text inside a black table on a page with a white background, for example, will appear to an engine to be hidden text.
Search engines ban redirects as well. A splash page with a redirect to your home page is automatic trouble.
Designers also need to be aware that search engine spiders cannot follow image map links; you must include HTML links if you want search engines to spider all your pages.
Framed sites need to use the noframes section of frameset pages if search engines are to see them.
Dynamically generated pages also present special problems to search engine spiders.
Off Page Criteria
To provide more sophisticated measures of relevance and to combat spamming, search engines are trying new technologies and criteria for ranking.Special weight is being given to index pages on some engines, undoubtedly as an anti-spamming measure.
Direct Hit is providing engines with measurements of site popularity, and Direct Hit results appear in various forms on many engines.
Link popularity and link relevance also are factored into many search engine rankings. Just as good personal connections help in real life, so do good quality links help in search. Google is notable for its emphasis on link relevance.
Building links to popular related sites will help your site be found on the Web in any case, and are a good investment of time. Look for business directories and other professionally oriented sites that will link to yours. Search terms contained in the link text can give you a search boost on those terms.
Search Engine Submissions
Submissions are important enough that you want to monitor them in process. Submit very carefully and submit by hand. Keep logs of submissions. Basic submission guidelines are found on all of the engines and directories.
Some engines will accept submissions of any page on your site. Others accept index page submissions only and will then crawl the site for other pages.Stick to the 12-15 major engines and directories, and avoid the free online submission services. Many major engines disregard submissions from these services, which are most often feeding sites that really do nothing but collect email addresses.
Inktomi, it should be pointed out, is the underlying search engine for a variety of major search sites, including HotBot, MSN Search, Snap, AOL, and Yahoo Web Pages. They all use Inktomi data in different ways. The best way to submit to Inktomi is via HotBot.
Detailed information about the major search engines and directories can be found on Search Engine Watch, at
www.searchenginewatch.com
. The site is an invaluable resource for anyone interested in search engines and search optimizing.
Read
Part Two of this series
Robert Charlton works in a variety of media and consults in Web site search engine optimizing and search engine-friendly design. His search optimizing clients include companies in the fields of publishing, wine, clothing, photography, psychological testing, telecommunications, and entertainment.
