In this article
Most website visits start when someone types a search into Google or another search engine. For your site to have any chance of succeeding, it needs to be visible to the search engines, or it won’t appear in the results. Search engine software known as a ‘crawler’ or ‘spider’ regularly scans websites and adds them to the search engine’s index. They then appear in the results when someone searches for a relevant keyword or phrase. There are a variety of reasons why your website might not be indexed properly. Consider the following tips to fix the problem and make your site more search engine friendly. Google Search Console’s Page indexing report is a great place to start: it shows which of your pages aren’t indexed, and why.
Pages Inaccessible Without a User-Submitted Form
Any section of your website which requires extra user interaction to access will not be indexed by the search engines. If you want the content of your site to be indexed, then it needs to be public. Search engine crawlers cannot submit forms or do anything else other than scan content which is visible to anyone. Any content or links that you want the search engines to see should be outside of any restricted section of the site. Likewise, search engine crawlers cannot see any content which is only accessible by performing a search.
Content in Flash, Plug-Ins and Scripts
Old plug-in technologies such as Flash and Java applets are no longer supported by modern browsers, and search engine crawlers could never read their content anyway. Content that only appears after complex JavaScript runs can also be harder for search engines to index. Make sure your important text and links are in the page’s HTML, where both visitors and search engines can see them.
Content Blocked by robots.txt
Robots.txt is part of the robots exclusion protocol (REP). It is a text file stored in the root folder of your website, which lets webmasters tell search engines which pages of their website they shouldn’t crawl. If your site doesn’t appear in any search results, check your robots.txt file to see whether anything important is blocked. There are also other ways to keep content out of search results, such as the robots meta tag (noindex), which gives you more control page by page.
Too Many Links on a Page
Search engines can follow many links on a page, but pages stuffed with hundreds of links look like spam and spread their value thin. If you have a page containing too many links, split it into several pages, ideally organizing your links into categories in the process, to make things easier for your visitors.