Not being indexed Solved/Closed

Question

Good evening,

I wanted to know how to prevent my website from being indexed by Google or any other search engine?
I'm asking this question due to the following issue:

When I type in Google the name of my domain, I come across one of the pages of one of my sites hosted under my domain.
I don't want it to appear in a Google search because the path to my pages appears ^^

Thank you for your help :)

~ Don't forget the "Resolved" tag when your issue is... resolved :) ~Configuration: Windows XP
Firefox 2.0.0.6

gryzzly · Accepted Answer

<META NAME="ROBOTS" content="noindex"> in all the pages to exclude -- One can forgive much to others, except having helped you. B. Werber

jérémy1664 · Answer

Hello

To avoid being indexed (weird), you need to put a few instructions in your site's index for the robots.txt (from Google)

Here are examples of robots.txt files:

* Exclusion of all pages:

User-Agent: *
Disallow: /

* Exclusion of no pages (equivalent to the absence of a robots.txt file, all pages are visited):

User-Agent: *
Disallow:

* Authorization of only one robot:

User-Agent: robotName
Disallow:
User-Agent: *
Disallow: /

* Exclusion of a robot:

User-Agent: RobotName
Disallow: /
User-Agent: *
Disallow:

* Exclusion of a page:

User-Agent: *
Disallow: /directory/path/page.html

* Exclusion of all pages in a directory and its subfolders:

User-Agent: *
Disallow: /directory/

kij_82 · Answer

Yes exactly, thank you for the response.
I had also found on the page about natural referencing, so I allow myself to complement the answer.

Here are the different values for the 'content' attribute:
- all: allows all robots to index, archive, etc. your pages
- noindex: prohibits indexing by all robots
- nofollow: prevents your pages/links from being followed
- noarchive: prevents your pages from being archived
- none: equivalent to 'noindex,nofollow'

Ex: <META NAME="ROBOTS" content="none,noarchive">

If your pages are already indexed on the net by Google or another robot, you will need to place a file named 'robots.txt' at the root of your site. In this file, you will put the following line:


# robots.txt 
User-agent: * 
Disallow: /

User-agent: allows you to specify which type of robot. '*' means all robots. If you only want to prohibit indexing on Google, replace '*' with 'googleBot'
Disallow: specifies which directory of your site should not be indexed.
You can therefore add as many lines as necessary for each directory to be prohibited as follows, for example:


# robots.txt 
User-agent: * 
Disallow: /toto/ 
Disallow: /mesPhotos/

The '/' in the first example refers to the current directory (where the robots.txt file is placed)

There you go :)

EDIT: My apologies Jérémi, I hadn’t seen your post ^^

However, I quote:

for robots.txt (for Google)

In fact, it’s the standard for any robot normally, so it works for others than Google.

Kyoshiro38 · Answer

Same, I don’t see anything strange about not wanting to be indexed.

Like, you develop a site for a client, you have a production version that needs to be indexed, and a development version where you can tinker and show the progress to the client. The development version has no reason to be indexed and shouldn’t be.

You might tell me that if no external link points to the development version, there’s no reason for Google to come here!!! That’s true, but it only takes one fool (and clients often are!!!) to put the link to the development version in a forum to talk about it to a friend, and boom, the bots sneak in by following the link!!!!

The theory of that brilliant algorithm called PageRank suggests that bots just follow links and wander from page to page. In practice, these nice little bots are far more intrusive than they should be!!! Once they've sniffed around your site, they scan, index, and cache everything they find.

I speak from experience when I say that documents placed in a directory of your website can be indexed and cached even if no link points to them!!!

Caution is essential, Big Brother is watching you!!!!

Searaphina · Answer

sometimes we put the site online but with a temporary domain that's why we don't want our site to be indexed (referenced by Google) to avoid falling into the case of duplicate content

Not being indexed

5 réponses

Samsung j5 unable to answer

Scam of the suvedene highway

Minecraft restore the cursor and the bar.

My ps3 keeps shutting down on its own...

Samsung series 4 led tv logo display...

How to enable bluetooth on a samsung tv?

Ctrl key issue

[metin2] sura ...

Freebox tv locked step 2 or network

Error 46 (brother dcp-150c) cleaning impossible