You too can be big brother: inurl:/view.shtml

If you’re using Google, someone might think you are evil. This article suggests that Google:

“….has quietly provided assistance to several U.S. intelligence agencies, such as the Central Intelligence Agency and Defense Intelligence Agency, as the U.S. prosecutes its war on terrorism. In addition, Google may be providing assistance to the National Security Agency.“

I have no idea if that is true or not, but it does sound plausable. Why wouldn’t those agencies tap into Googles knowledge store? Perhaps, at this very moment, several young gentlemen with crazed glints in their eyes, are searching Google maps, furiously g-mailing each other, looking up do-it-yourself sites of dubious distinction, all while making use of the damned technology of the infidel.

They’re probably also finding a lot of Wikipedia in the results.

The trouble is, they could just as equally be regular people looking up information, who just happen to conduct a few search queries that just happen to appear on the “suspicious” list.

How thin is the line?

Anyone would think there’s a turf war going on in Belgium, that could spread even wider. Or something

Google outline their approach to content:

“Protecting content owners’ rights, respecting their wishes, helping to reward them for their creative endeavors – these are the primary principles that guide Google’s approach. We believe it’s the best way to make the world’s information universally accessible and useful.“

No surprises there.

I do believe that Google are, generally speaking, acting appropriately, however I don’t think they’re quite as divorced from content provision as they like to make out.

If content is defined as the authorship of words on a page, or a piece of video, then - no, Google doesn’t do content. But is content more than that? If you scan across Google News, it looks like the front page of a newspaper. Is the front page of a newspaper not content? You can scan across Google News, get a good idea of what is going on, and not have to click through to the source. I guess it could be argued that the value is mostly in the aggregation and categorisation.

A question: does the author who publishes an html document on a web server, give a search engine the right to index that document simply by virtue of publication?

Sitelinks are those sub-section of nested links you sometimes see in search results.

Like this:

Google says:

“Our systems analyze the link structure of your site to find shortcuts that will save users time and allow them to quickly find the information they’re looking for. We only show Sitelinks for results when we think they’ll be useful to the user. If the structure of your site doesn’t allow our algorithms to find good Sitelinks, or we don’t think that the Sitelinks for your site are relevant for the user’s query, we won’t show them. At the moment, Sitelinks are completely automated. We’re always working to improve our Sitelinks algorithms, and we may incorporate webmaster input in the future.“ 

Matt Cutts made a comment in this thread over at Threadwatch.org in relation to the problem with getting accurate data using the site: command in Google:

Anyway, I believe I’ve said that 72.14.207.104 is the most likely to return accurate site estimates because it has some new infrastructure, and that I hoped the new infrastructure would be at all data centers by the end of summer, which would be about another month. Bear in mind that we sometimes take data centers out of the rotation for various reasons, so even a single IP address 72.14.207.104 can hit different data centers at varying times. And different data centers will show different results estimates, which accounts for why you see different things when you try different IP addresses”

You’d have to have been living under a rock not to notice that the best “search engine optimized” site on the entire planet is: Wikipedia.

The thing is everywhere. For just about any generalist search phrase you care to name, there lies Wikipedia - a prime example of how links are infinitely more valuable than on-page factors

There is a downside.

If people consider Google an authority, and many do, and if people consider Wikipedia an authority, and many people do, are we in danger of promoting a homogenized world view?

Dave Winer:

“In the web before Wikipedia, every point of view had a chance, but Wikipedia tends toward centralization, toward one or two views prevailing, those that are represented by people who are willing to maintain a presence on Wikipedia. This what I’m not comfortable with.“

The ever helpful, cool-cat Matt Cutts explains about reinclusion requests. More importantly, he introduces us to one of his cats.

And here’s mine:

Rule #1 of blogging is don’t post pictures of your cat. Or write about what you had for breakfast. Yogurt (Strawberry). Fruit Muffin. Coffee (black).

You’ll get none of that here.

Anyways, if you suspect your website has been banned from Google, you can put in a reinclusion request. Matts guidelines for reinclusion are posted here.

In short, tell them what you did, fix it, tell them you’ve fixed it, and tell them that it won’t happen again. Can’t get fairier than that. Yahoo! & MSN should do the same.

Are having problems getting your content indexed by Google?

Google offer a helping hand, in the form of Webmaster Central, a collection of webmaster advice and tools, including Google Sitemaps.

“The Sitemaps protocol remains unchanged and Sitemaps submission mechanisms and reporting is still available from the Sitemaps tab. We’re also renaming our blog and Google Group to be more broadly focused on webmaster topics”.

Hurrah!

They’ve made a few changes to Sitemaps too, which you can read about on their official Webmaster Central blog.

BTW, the Google Blog has a jolly useful list of all the Google Blogs. So I’ve knicked it…erm…”contextualized it”, and here it is

• AdWords API Blog
• Blogger Buzz
• Google AJAX Search API
• Google Analytics Blog
• Google Base Blog
• Google Blog
• Google Blog (Brasil)
• Google Blog (China)
• Google Blog (Italia)
• Google Blog (Japan)
• Google Blog (Korea)
• Google Blog (Mexico)
• Google Checkout
• Google Code
• Google Enterprise Blog
• Google Maps API Blog
• Google Reader Blog
• Google Research
• Google Talkabout
• Google Video Blog
• Google Web Toolkit Blog
• Inside AdSense
• Inside AdSense (Dutch)
• Inside AdSense (German)
• Inside AdSense (Spanish)
• Inside AdWords
• Inside Google Book Search
• Inside Google Desktop
• Inside Google Sitemaps

Matt Cutts offers a few tips on how to handle Googlebot. These tips will help ensure your site is indexed correctly.

From Matt Cutts Blog:

• At a site or directory level, I recommend an .htaccess file to add password protection to part of a domain.
• At a site or directory level, I also recommend a robots.txt file.
• At a page level, use meta tags at the top of your html page.
• At a link level, you can add a nofollow tag on the granularity of individual links to prevent Googlebot from crawling individual links (you could also make the link redirect through a page that is forbidden by robots.txt).
• If the content has already been crawled, you can use our url removal tool.

There are also a few curious pieces of information in the comments, namely that Google has “gotten better” at crawling Javascript links. I’ve also noticed that Google has “gotten better” at crawling scripted cgi-links, which unfortunately can earn you a dup content penalty.

Google have added a cool new gadget for personalized search pages. It’s called search history trends and provides statistics about your search patterns.

One really interesting chart is the top gaining queries related to your searches, which reveals “movers” in your areas of interest, although mine was a little out. “US Golf Open”?

I guess it gets better as you go along….”