l i n u x - u s e r s - g r o u p - o f - d a v i s
Next Meeting:
July 7: Social gathering
Next Installfest:
Latest News:
Jun. 14: June LUGOD meeting cancelled
Page last updated:
2011 May 25 13:52

The following is an archive of a post made to our 'vox-tech mailing list' by one of its subscribers.

Report this post as spam:

(Enter your email address)
[vox-tech] how to modify .htaccess to prevent wget or the likesfrom downing my site?
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[vox-tech] how to modify .htaccess to prevent wget or the likesfrom downing my site?

Hello all:

I first asked this question to the support of my web host, and they
redirected me to this link:

and the snippet on that page looks like:

SetEnvIfNoCase User-Agent "^Wget" bad_bot

<Limit GET POST>
   Order Allow,Deny
   Allow from all
   Deny from env=bad_bot

I copied and pasted it to the .htaccess under /public_html. Still, I
am able to use this command to fetch my site:

wget --wait=20 --limit-rate=20K -r -p -U Mozilla www.my_iste.com

I noticed the "Wget" on the above snippet has a capital "W" therefore
I changed it, no difference thou.

However, if I  tried the same wget with a slight change in the command
line (without " -U Mozilla ")

 wget --wait=20 --limit-rate=20K -r -p www.my_site.com

I get this:

--2011-05-25 14:30:36--  http://www.my_site.com/
Resolving www.my_site.com... xxx.xx.xxx.xx
Connecting to www.my_site.com|xxx.xx.xxx.xx|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2011-05-25 14:30:37 ERROR 403: Forbidden.

On the same page
(http://www.webhostingtalk.com/showthread.php?t=437549), I noticed a

wget -U "Mozilla/4.03 [en] (X11; I; SunOS 5.5.1 sun4u)"

Use of this option is discouraged, unless you really know what you are doing.


Now I have three questions:

1. Why didn't the code in .htaccess prevent the downloading? Did I
miss something?
2. Do we have other tools acting like wget, how can we prevent them
all from downing the site content?
3. If someone is downloading, can we have some log file that can
expose the downloader's info?

Thanks a lot!
vox-tech mailing list

LUGOD Group on LinkedIn
Sign up for LUGOD event announcements
Your email address:
LUGOD Group on Facebook
'Like' LUGOD on Facebook:

Hosting provided by:
Sunset Systems
Sunset Systems offers preconfigured Linux systems, remote system administration and custom software development.

LUGOD: Linux Users' Group of Davis
PO Box 2082, Davis, CA 95617
Contact Us

LUGOD is a 501(c)7 non-profit organization
based in Davis, California
and serving the Sacramento area.
"Linux" is a trademark of Linus Torvalds.

Sponsored in part by:
EDGE Tech Corp.
For donating some give-aways for our meetings.