ASP.NET sitemap , how important?

Question

I am finished with my website, or so I thought. I don't have a sitemap and by some miracle I've missed the entire concept of a sitemap, didn't even know it was a thing. Big shoutout to my CS teachers I guess.

I've been reading up on it and it seems quite complicated to generate sitemap dynamically, which I have to do since my page is basically just a big database where you search using parameters.

How important is this for Google search engine relevance etc? For example, if someone searches for my website on google like this "www.mySite.com food". If I have a category in my database called "food" and indeed if someone uses the search on my site they will find a food-category, but will Google know this? Will Google find "www.mySite.com/Find?Result=food"?

EDIT: Is it even correct to use Web.Sitemap that you can create in ASP.NET or should I use a file called sitemap.txt?

Answer 1

Site Map Types

There are 3 different kinds of "site maps".

1. A page that has links on it that assists users by giving them a complete outline of the site structure. This sometimes, but not always, follows the site hierarchy. Also, it is usually limited to major categories of the site. The detail pages are not normally listed.
2. A system in ASP.NET that helps to build menus, breadcrumb trails, and other navigation features of the site. This is driven by a provider that by default uses .sitemap XML files.
3. An XML document that provides links to the site's pages along with information about when the page was last updated, how important it is to update, etc. to assist with indexing the site in search engines.

A Brief History

If you look at the history, it makes parsing this information a bit less confusing. The above list is in chronological order. "Site map" was a term originally given to provide a user page that linked to the major sections of the web site. Microsoft decided to again use the term "site map" to refer to their navigation system for ASP.NET 2.0. Some years later, the major search engines teamed up to provide an XML specification for providing URL information about sites, and they unfortunately overloaded the term "site map" once again to describe them.

One of the consequences of this history is that ASP.NET has no built-in support for XML sitemaps, which it sounds like you are trying to make based on your question. This is primarily because Microsoft's implementation hasn't changed (much) since ASP.NET 2.0 and it was written before that specification came to fruition. So, using Microsoft's ASP.NET navigation system will do you no good for the purpose of generating an XML sitemap.

SEO Considerations

Do note however that all 3 of these items are important for SEO.

1. A user site map helps with search engine indexing because it provides inner site links to the major areas that need to be indexed.
2. Menus and breadcrumb trails are important to link all of your content pages to each other, which helps to identify relevance between different parts of the site.
3. XML sitemaps provide search engines direct URL information and help to reduce search engine traffic on the site and to help ensure there are no important URLs that the search engine fails to consider for indexing.

So you may want to take each of these into consideration, although strictly speaking all of them are optional.

SEO is about first providing good content, and then doing the required steps so that content is accessible. Each of the above should be considered a step forward for SEO and if you really care about search engine placement, you should do all that is possible, including all 3 of these steps.

XML Sitemap Recommendation

Actually, the XML sitemap specification is pretty simple for small sites under 50,000 pages. So it generally doesn't take much effort to put them together. You may want to consider doing this yourself.

Most of the open source implementations are not very scalable because they all load the entire set of data into memory at once to generate the XML instead of streaming it. Also, most of them don't really account for the fact that you might have dynamic data in multiple tables that all contribute to the URLs of the web site. They usually leave it up to you to provide paging. Most of the existing implementations also do not provide the specialized content types offered by Google.

I have recently created an implementation that does all of these things. It has not officially been released, and it is currently part of the MvcSiteMapProvider project. We have plans to separate the XML sitemap functionality into a different assembly (and separate NuGet package) so it can be used without referencing MVC but for now MVC is a requirement (unless of course you want to grab all of the relevant types and compile them into a separate DLL yourself). An ASP.NET project with MVC in it will still run just fine, so this isn't that much of an issue for most projects.

There isn't any documentation at this point, but I have created a quick post showing how to wire it up in MVC, including a demo application.

For ASP.NET, you could either use the MVC implementation (if you install the MvcSiteMapProvider, it will install MVC into your project anyway), or you could just use an ASP.NET page.

In the second case, the configuration would look similar with a few exceptions.

1. You don't need to use the XmlSitemapFeedRouteRegistrar class.
2. You (of course) don't need to use the Controller, and you would replace that with an ASP.NET page, like this:

XmlSitemap.aspx

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="XmlSitemap.aspx.cs" Inherits="XmlSitemap" %>

XmlSitemap.aspx.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using MvcSiteMapProvider.IO;
using MvcSiteMapProvider.Xml.Sitemap;
using MvcSiteMapProvider.Xml.Sitemap.Configuration;

public partial class XmlSitemap : System.Web.UI.Page
{
    private HttpContextBase HttpContext
    {
        get { return new HttpContextWrapper(System.Web.HttpContext.Current); }
    }

    private int PageNumber
    {
        get
        {
            var pageString = HttpContext.Request.QueryString["page"];
            if (!string.IsNullOrEmpty(pageString))
            {
                int page;
                if (int.TryParse(pageString, out page))
                {
                    return page;
                }
            }
            return 0;
        }
    }

    private string FeedName
    {
        get
        {
            var feedName = HttpContext.Request.QueryString["feedName"];
            if (!string.IsNullOrEmpty(feedName))
            {
                return feedName;
            }
            return "default";
        }
    }

    protected void Page_Load(object sender, EventArgs e)
    {
        // Build feeds for XML sitemap
        var xmlSitemapFeedStrategy = new XmlSitemapFeedStrategyBuilder()
            .SetupPageNameTempates(templates => templates
                .WithDefaultFeedRoot("XmlSitemap.aspx?feedName={feedName}&page={page}")
                .WithDefaultFeedPaged("XmlSitemap.aspx?feedName={feedName}&page={page}")
                .WithNamedFeedRoot("XmlSitemap.aspx?feedName={feedName}&page={page}")
                .WithNamedFeedPaged("XmlSitemap.aspx?feedName={feedName}&page={page}"))
            .AddDefaultFeed()
            .AddNamedFeed("google", feed => feed.WithContent(c => c.Image().Video()))

            // Optional - add news feed (will be at ~/XmlSitemap.aspx?feedName=news)
            .AddNamedFeed("news", feed => feed.WithContent(c => c.News()))

            // Optional - add mobile feed (will be at ~/XmlSitemap.aspx?feedName=mobile)
            .AddNamedFeed("mobile", feed => feed.WithContent(c => c.Mobile()).WithMaximumPageSize(10000))
            .Create();

        var xmlSitemapFeed = xmlSitemapFeedStrategy.GetFeed(this.FeedName);

        if (xmlSitemapFeed != null)
        {
            var outputCompressor = new HttpResponseStreamCompressor();
            var response = HttpContext.Response;

            response.Clear();

            // Output content type
            response.ContentType = "text/xml";

            using (var stream = outputCompressor.Compress(HttpContext))
            {
                if (!xmlSitemapFeed.WritePage(this.PageNumber, stream))
                {
                    response.Clear();
                    //Return 404 not found
                    response.StatusCode = (int)System.Net.HttpStatusCode.NotFound;
                    response.StatusDescription = "Page Not Found";
                }
            }

            response.End();
        }
    }
}

The above configuration actually creates 4 different XML sitemap feeds, mostly to adhere to the specification of these specialized types.

1. Default - /XmlSitemap.aspx - A sitemap that can be used with any search engine. This URL needs to be in your robots.txt file as per the XML sitemap specification.
2. Google - /XmlSitemap.aspx?feedName=google - This sitemap includes the same URLs from Default, but it also includes the Image and Video specialized content. Since incorrectly coded crawlers may crash if they encounter any of Google's specialized types, I have separated them into a different feed. You would need to manually submit this URL to Google through their webmaster tools.
3. News - /XmlSitemap.aspx?feedName=news - This is for Google's news sitemap functionality, which needs to be submitted per their specifications.
4. Mobile - /XmlSitemap.aspx?feedname=mobile - This is for Googles mobile sitemap functionality, which needs to be submitted per their specifications.

Note that it is also possible to use routing in ASP.NET to make these URLs nicer. You just need to ensure that the name templates are specified above the same way that the URLs are generated by routes, putting the placeholders for {feedName} and {page} in the appropriate place in the URL.

Do be aware that according to the XML sitemap specification, an XML sitemap cannot contain any URLs for directories above its virtual directory. So, it is best to ensure your route does not contain any / characters (meaning it applies to the whole site, not a subdirectory).

Answer 2

First, no. The asp.net sitemap isn't what you would really use for Google. That's more for informational purposes. There is a sitemap file that most search engines accept as a way to better understand your site. There are lots of free and paid services that will generate it for you. Just Google "sitemap generator" and you'll find a huge variety. These will spider your site, find the links, organize them, and give you the files in an XML sitemap for major search engines.

Google isn't going to go and find categories so it won't know about Food unless you link to it somewhere. You may also want to look at using url rewriting instead of querystring variables as it improves SEO. Your url would be: www.mysite.com/Find/Food, which looks friendlier to search engines.

In short, the asp.net sitemap is nice, but intended to be consumed by other data components to create things such as a menu or a visual sitemap for users and not the sitemap definition file used by a search engine.