l i n u x - u s e r s - g r o u p - o f - d a v i s
Next Meeting:
July 7: Social gathering
Next Installfest:
Latest News:
Jun. 14: June LUGOD meeting cancelled
Page last updated:
2011 May 25 14:13

The following is an archive of a post made to our 'vox-tech mailing list' by one of its subscribers.

Report this post as spam:

(Enter your email address)
Re: [vox-tech] how to modify .htaccess to prevent wget or the likesfrom downing my site?
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [vox-tech] how to modify .htaccess to prevent wget or the likesfrom downing my site?

On Wed, 2011-05-25 at 13:51 -0700, Tony Cratz wrote:
> On 05/25/2011 11:50 AM, Hai Yi wrote:
> > I first asked this question to the support of my web host, and they
> > redirected me to this link:
> > http://www.webhostingtalk.com/showthread.php?t=437549
> > 
> > and the snippet on that page looks like:
> > 
> > 
> > SetEnvIfNoCase User-Agent "^Wget" bad_bot
> > 
> > <Limit GET POST>
> >    Order Allow,Deny
> >    Allow from all
> >    Deny from env=bad_bot
> > </Limit>
> 	From reading this message and latter ones I have to ask, would
> 	it be better to do a deny from all and allow only the browser
> 	you want? At times it is easier to do things the reverse way
> 	of doing things as it can be must simplier to set up.

While allowing only user agents beginning with "Mozilla" would be an
attractive option, there are two big problems:

First, this wouldn't allow search engines to index his site. He could
allow specific search engines based on their user-agents, but then he'd
risk missing the next new big search engine if its indexing bot had a
brand new user-agent that wasn't on his white list.

He'd probably forget to allow text-based browsers like lynx, Links, and

vox-tech mailing list

LUGOD Group on LinkedIn
Sign up for LUGOD event announcements
Your email address:
LUGOD Group on Facebook
'Like' LUGOD on Facebook:

Hosting provided by:
Sunset Systems
Sunset Systems offers preconfigured Linux systems, remote system administration and custom software development.

LUGOD: Linux Users' Group of Davis
PO Box 2082, Davis, CA 95617
Contact Us

LUGOD is a 501(c)7 non-profit organization
based in Davis, California
and serving the Sacramento area.
"Linux" is a trademark of Linus Torvalds.

Sponsored in part by:
Sunset Systems
Who graciously hosts our website & mailing lists!