I found this post when searching for I just got a call from my hoster because my bandwith usage is insane. I take it, most of you on here have blocked this site, and it's not really necessary? Create a file in your account and insert the following lines: This will disallow Yandex robot to access any file under your account. Yandex has spiked my bandwidth with a ridiculous number of visits (far, FAR more than Google bots EVER make). Yandex has been hitting 404's for WEEKS, multiple times a day. I am NOT saying Yandex is trying to hack my sights. The point is, their crawler is woefully inefficient, displays no respect for my rules, and looks for all intents and purposes like it's acting like a hacker.BTW, there are bad robots which ignores file in which case, you have to manually block the IPs of these servers either using the method stated above by "rustelekom" OR block the IPs server wide using any 3rd party firewall or iptables. I came for the same reason as the above poster, looking for a way to block Yandex. " The ONLY way to keep them out is through .htaccess, and that's what I'm looking for too. Too much Internet crime originates in Russia and several other Easter European countries. Before I tried blocking them in robots.txt, they were ignoring all the rules there, exposing much of my content to security risks. That is exactly why there are so many forum posts all over the web from webmasters trying to figure out what it is (it doesn't occur to most that it could be a legitimate search engine, because it doesn't ACT like a legitimate search engine). There is another, and what I believe to be a very sound reason to block them. Also some examples of repeatedly crawled 404's would be very helpful. Best regards, Yaroslav With all due respect, it does not visit at all when it comes to my sites.The scammers are using their images without their knowledge or permission to deceive their victims and steal their money.The scammers are using their images without their knowledge and permission to deceive their victims and scam them out of money......It could very well be that asking them to remove your site from their lists, actually works. but that takes time I simply do not have, and I have other matters that need attention.) For all I know, you could be the very person who hacked my sites to begin with.But I have no reason to believe it, and I already have a way to deal with them that works. I'm not saying you are though, and the likelihood is very low. BTW: The "correct" way to do things is by doing what works! I do use When a robot breaks the rules in there, they are banned. Now, that is a method I haven't yet seen posted anywhere. When a robot, any robot, doesn't even bother to look at the rules, that's a BAD robot.
BTW, there are bad robots which ignores file in which case, you have to manually block the IPs of these servers either using the method stated above by "rustelekom" OR block the IPs server wide using any 3rd party firewall or iptables. " When robots do not adhere to the rules in robots.txt, as far as I am concerned, they are not playing fair, or nice. They look for them a few times, over the course of a week or so, presumably to see if they reappear, then quit looking. A HUGE number of their "hits" were 404 before I blocked them. Why, if the robot has already determined multiple times that a file is not there, does it keep looking for it over and over again? Yandex is not just visiting way too many times per day, they are crawling files that haven't even been on the site since BEFORE they began crawling it.I want to find a man who will treat his woman with respect, who will never let her down and will be responsible and firm in his goals and aims. They search through the social networks and dating sites databases, to see if a person with such email is registered somewhere.RESEARCH YOUR SCAMMER, FIND OUT WHO IS BEHIND THE EMAIL Tanya Member Name: tanyadasova Member ID: 024005 Caucasian / White, 36, Single Zhitomir, , Ukraine Personal Details Gender: Female Age: 36 (15-Mar-1980) Race: Caucasian / White Marital Status: Single Children: 0 Religion: Christian / Orthodox Occupation: Other Education: University Languages: English (Spoken and Written) Russian (Fluent) Ukrainian (Fluent) I am a polite, sincere and honest woman. I know that probably I am not a perfect woman and I have some demerits of my own.