-----------------------------------

avalynmarie

Sun Jun 24, 2007 7:38 pm


Robot.txt file

-----------------------------------

Okay, I keep reading about this robot.txt file and as I understand it, it tells search engines which part your site not to crawl. Is that correct? Also, what is the exact code that needs to be copied into the website editor? I keep finding long-winded explainations that I don't completely understand, what I really need is code that I can copy and paste.


-----------------------------------

b0b0

Tue Jun 26, 2007 1:03 pm



-----------------------------------

Hi,

well... here is an example for my PC .... I'm using easyphp (apache + mysql) to create my websites...

I noticed at some point google was visiting some websites on my pc ! :D (well some link must have been followed online that redirected to my no-ip account on my pc)
Anyway ...

I created a 
robots.txt file at the root of my website:
Containing :

User-agent: * 
Disallow: /b0.b0.free.fr/


This tells ALL the robots, spyders, crawlers to NOT visit the folder /b0.b0.free.fr/
Now bear in mind that some bad robots won't even read that file ... they will crawl the whole website ... :(

ps : you don't need a website editor for this .... the windows notepad is enough to create this txt file that you can later upload on your server :)

b0b0


-----------------------------------

avalynmarie

Tue Jun 26, 2007 2:09 pm



-----------------------------------

how do I upload it to my server? I've only ever used the editor provided by my host, GoDaddy.


-----------------------------------

b0b0

Wed Jun 27, 2007 8:57 am



-----------------------------------

Don't you have a FTP access to your webserver ?
If yes, then just upload robots.txt in the root of your website... 

If you don't have ftp access, there must be another way for you to upload a txt file the same way you uploaded your images on your website, but not sure on how godaddy works :/


b0b0