I was a panelist at a local conference on Internet marketing here in Iceland last Friday.

As you can imagine, a lot of the time was spent on talking about marketing using search engines. Both how to use paid for placement (i.e. ads) and how to optimize pages to rank better in the natural search results.

The latter, usually called “search engine optimization” or just SEO has a somewhat lousy reputation. More or less everybody that owns a domain has gotten a mail titled something like: “Top 10 ranking guaranteed” from people that claim that they can work some magic on your site (or off it) and it will get you to the top of the natural search engine results for the terms that you want to lead people to your site.

Needless to say, these people are fraud. If the spam isn’t enough of a giveaway there are two additional reasons. First of all, all the major search engines guard their algorithms fiercely and change them frequently, exactly to fight off people that are trying to game their systems in this way. Secondly they will probably choose an oddball phrase (see the story of Nigritude Ultramarine) that nobody else is using and have you put that on your site. Needless to say, it will get you a top 10 ranking ‘coz you’re the only one using it.

Additionally some of these “respectable professionals” have so called link farms that are in fact huge infinite loops of pages linking to each other in order to give the impression that any one of these pages is a highly popular one because it has so many incoming links. The link farm pages (here is an OLD blog entry I did on that) are usually machine generated and make little or no sense to humans, but may trick the search engines – for a while. When the search engine engineers find out however, they will get in foul mood and are likely to punish or even ban all the pages involved from their search engines, and then you’re better off without them in the first place!

There is a range of other tricks that bad SEOs use, but the results from using them are usually the same: It will rank you high for a while, and then have the opposite effect.

On the other hand, there are a series of things that you should do to make it easier for search engines to understand what your site is all about, without playing dirty. Most of these things are exactly the same things as make the page more usable for humans: Descriptive page titles, informative link texts, proper use of the terms you want to link to your site and last but not least good and relevant information on your subject. If you have a highly usable and informative site, it will lead to people linking to you as an information source and that will give you quality, incoming links from respectable sites – one of the most important things for ranking well, especially at Google.

Enough about that – others have written much better introductions to search engine marketing and a lot of respectable people make a living out of giving healthy tips and consult on these issues.

The main point is that in the search engine world, there is a constant fight between the bad SEOs and the engineers at the search engine companies.

During dinner, after the conference – I had a very interesting chat with one of the speakers, who expressed his concern that “search was the next spam”, meaning not that people will try to game the search engines (they have done so sinvce way back in the 90s), but that they will get so good at it that it will start to hurt the search business in the same way as email spam has hurt email.

As mentioned before, there are two ways to get exposure in search engine results: paid for placement and natural search results. With paid search estimated a 2.2 billion dollar industry last year, there is a lot of incentive for a lot of clever (yet maybe a little immoral) engineers to try their best to rank high in the natural results. The natural results are a lot more likely to get clicked and therefore more valuable than the paid ones – anybody want to estimate a number?

And the evidence is out there. Even Google sometimes gives me results that are obviously there for the “wrong reason”. Other engines do it more frequently. And even with the “non-spam” results there has been talk recently about the majority of search results being commercial – even in fields where you would expect to get both commercial and non-commercial results.

The paid for placement results are of course commercial by nature – no problem there as long as they are clearly marked as such. The natural results on the other hand, should reflect the best information on the web – commercial or not. The problem is that commercial sites are usually made by professionals and by now, most of them pay – at least some – attention to optimization that will help them getting higher ranking in the search engines. Not necessarily the bad tricks, but just the good ones. And therefore the majority of the natural results is and will be commercial too.

Search engine indexes, populated by machine bots – engineering marvels as they are – simply cannot make the needed distinction here. The ones that can tell what people will find to be good results are – you guessed it – other people.

Human information will become increasingly important in the fight against spam and to keep the search engine results free of commercial bias. Human information was what created Yahoo! in the beginning. This was also the brilliance of Google’s page rank and link analysis when they began – they were tapping into human information, links that people (webmasters) created were considered a “vote” and meta information about the page they linked too. This is what search engines saw as a major asset in the blogging community and this is why humanly created indexes like the ones that are constantly growing now at bookmarking services such as my Spurl.net, del.icio.us and LookSmart’s Furl will become major assets in the search industry in the coming months and years. And with a decent reputation / trust system (think Slashdot), it will be relatively easy to keep the spammers out – at least for a while.