Forum

Notifications
Clear all

High Resource Usage on Lunarpages

Talullah
(@talullah)
Trusted Member

Hi all,

I have installed eFiction for a friend whose site is hosted at Lunarpages ( http://www.ofelvesandmen.com/). The new eFiction site has not been widely divulged, as she's planning a grand opening, but it's not a well kept secret either and we've had a few 'Rolex reviews' if you know what I mean.

A few days ago some seven or eight people registed (with odd, spammish names), but no new reviews or other activity was detected. Shortly after that, we were contacted by Lunarpages saying we were exceeding our quotas and our resource usage was too high. The kept insisting on this and now they are threatening to shut the site down in 72 hours if we don't fix it.

They do provide some advice, which we will follow, but I was wondering if anyone else has been affected by something like this and what did you do. Also, the owner has not rulled out changing hosts - any advice? I have a site at Aplus and I love them but I don't really know the market - my research dates back a decade... 🙂

Anyway, I'll paste below what they say. We'd be very, very grateful for your insight.

Cheers,
Talullah

Hello,

Thanks for contacting us, As per your request we have scanned complete account and result is clean, no exploited or bad file is there.

We've performed another audit of your account today and it's, unfortunately, still exceeding the limits on our server.

Stats for 28 Sep 2016:
---------------------------------
CPU Usage - %0.19
MEM Usage - %7.80 HIGH
Number of MySQL procs (average) - 0.02
Top Process %CPU 11.00 /usr/bin/php /home/ofelve2/public_html/oeam/search.php
Top Process %CPU 0.80 cpaneld - serving 69.178 --llu1474976725 --listen10,3,4,5,6,7,8,9 --accepted11
Top Process %CPU 0.60 httpd [ofelvesandmen.com] [/oeam/browse.php?typeseries&charid425&letI&letZ]

Stats for 27 Sep 2016:
---------------------------------
CPU Usage - %0.16
MEM Usage - %5.25 HIGH
Number of MySQL procs (average) - 0.00
Top Process %CPU 20.00

Top Process %CPU 1.50 httpd [ofelvesandmen.com] [/Stories/A/Aglarien/WhatTheHeartSees.html]
Top Process %CPU 0.70 httpd [ofelvesandmen.com] [/Stories/A/Alex/Tolkien/I_P/L/LordCelebornsBook_Series/06_L]

Domlogs:
========================
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:56:23 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Lindir-Twins.htm HTTP/1.1" 200 4440 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:163.172.66.93 - - [28/Sep/2016:06:56:36 -0700] "GET /oeam/browse.php?type=series&charid=227&let=F&let=P HTTP/1.1" 200 11717 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.1; + http://ahrefs.com/robot/)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:56:38 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Luthien-.htm HTTP/1.1" 200 3725 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:180.76.15.136 - - [28/Sep/2016:06:56:43 -0700] "GET /Stories/A/ArielTachna/PriceOfPride/PriceOfPride36.htm HTTP/1.1" 200 17654 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; + http://www.baidu.com/search/spider.html)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:56:54 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Maedhros-.htm HTTP/1.1" 200 6537 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:57:02 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Maedhros-Maglor.htm HTTP/1.1" 200 4921 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:163.172.66.164 - - [28/Sep/2016:06:57:06 -0700] "GET /oeam/browse.php?type=series&charid=64&let=I&let=Other HTTP/1.1" 200 11805 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.1; + http://ahrefs.com/robot/)"
/etc/httpd/domlogs/ofelvesandmen.com:163.172.65.132 - - [28/Sep/2016:06:57:13 -0700] "GET /oeam/viewuser.php?action=favlist&uid=16 HTTP/1.1" 200 9847 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.1; + http://ahrefs.com/robot/)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:57:13 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Maedhros-Oropher.htm HTTP/1.1" 200 4047 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:51.255.65.92 - - [28/Sep/2016:06:57:15 -0700] "GET /Stories/A/Amber/Struggles.htm HTTP/1.1" 200 14166 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.1; + http://ahrefs.com/robot/)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:57:23 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Maedhros-Other.htm HTTP/1.1" 200 5545 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:57:34 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Maeglin-.htm HTTP/1.1" 200 4646 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:57:53 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Maglor-.htm HTTP/1.1" 200 6410 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:136.243.36.86 - - [28/Sep/2016:06:58:12 -0700] "GET /StoriesByFandom/Tolkien/StoriesByCharacter/H-Z/Maglor-Nimrodel.htm HTTP/1.1" 200 3743 "-" "Mozilla/5.0 (compatible; BLEXBot/1.0; + http://webmeup-crawler.com/)"
/etc/httpd/domlogs/ofelvesandmen.com:208.93.1.33 - - [28/Sep/2016:06:58:12 -0700] "GET /rss.xml HTTP/1.1" 304 - "-" "LiveJournal.com (webmaster@livejournal.com; for http://www.livejournal.com/users/oeam_updates/; 8 readers)"
=============================================
From the above logs, you can see that your website is being hit by bots. In order for your website to be found by other people, search engine crawlers also sometimes referred to as bots or spiders, will crawl your website looking for updated text and links to update their search indexes with.

When a search engine crawls a website, it requests the robots.txt file first and then follows the rules within.

Edit or create robots.txt file
--------------------------------------------------
The robots.txt file needs to be at the root of your site. If your domain was example.com it should be found:

You can also create a new file and call it robots.txt as just a plain-text file if you don't already have one.

Search engine User-agents
----------------------------------------------------
The most common rule you'd use in a robots.txt file is based on the User-agent of the search engine crawler.

Search engine crawlers use a User-agent to identify themselves when crawling, here are some common examples:

Googlebot
Yahoo! Slurp
bingbot

Common search engine User-agents blocked:
----------------------------------------------------
AhrefsBot
Baiduspider
Ezooms
MJ12bot
YandexBot

Disallow one particular search engines from crawling website:
----------------------------------------------------

You can disallow just one specific search engine from crawling your website, with these rules(like the one hitting your website):

User-agent: MJ12bot
Disallow: /

Disallow all search engines but one:
----------------------------------------------------

If we only wanted to allow Googlebot access to our / directory, and disallow all other bots we could use:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

When the Googlebot reads our robots.txt file, it will see it is not disallowed from crawling any directories.

If you are concerned with blocking bots will affect your website ranking, we would recommend you try to lower the rate of those bots crawling instead.

For Google bots, you can use Google Webmaster tool to reduce the rate of Google bot crawling. You can use Google Webmaster Central Tool at https://www.google.com/webmasters/sitemaps/ to reduce the rate of crawling a domain or you can use the following link https://support.google.com/webmasters/answer/48620?hl=en&ref_topic=2371325 to change the Google crawl rate.
====================
We suggest you please have a look at your incoming traffic and if the IP addresses are not legit to you please block them.

or it may just be that the site is receiving that high volume of traffic.

If you do need more resources, you might want to consider one of our VPS or Dedicated plans where you will have more control over your server and limits:

http://www.lunarpages.com/hosting/enterprise/dedicated/linux
http://www.lunarpages.com/hosting/enterprise/cloud/public
-----------------------------------------
Please also check if your scripts are optimized and upto date under the your account . We would suggest to please look into the script's functionality and update it so such issues does not reoccur.

However, it is also possible that the high CPU usage is unintentional and caused by some other factor. Below are some common software malfunctions that can cause high CPU usage and can be fixed without requiring an upgrade. It's worth checking into these to see if any are the reason for the account's high usage of CPU:

1. Regarding the frequent CRON jobs -- only run them as often as you absolutely need to! Check any cron jobs set up in cPanel and make sure they are not being run more frequently than every 30 minutes or less. Disable any database or other backups or other maintenance jobs that are set up, as these backups can be very CPU intensive.

2. Check your site for old installations (Fantastico or Softaculous) that have been left "in the wild". Check these for spam postings that look automated. If any are found, upgrade the application to the latest version (or remove it if it isn't in use), and make sure to setup Captcha, or comment approval, to eliminate automated posting. Check both your main domain (public_html) and the directories for any add-on domains that may have been set up.

3. Check web logs or statistics program (Webalizer in cPanel) for traffic. See what pages are getting the most traffic and where that traffic is coming from to look for any patterns that do not make sense.

4. Search engine spiders -- please make sure to have a robots.txt with the following line:
Crawl-delay: 30 or higher

5. Hacked account -- please keep all software up to date and use a secure password!

6. Check to make sure any MySQL databases are optimized with efficient queries

7. Check the PHP version being used in the PHP Configuration (cPanel) -- trying using the the last version if compatible with your software, for optimal performance.

Please note that this is a very important matter and it requires your (or your developer’s) immediate attention, if the situation will not improve over a 72 hours time frame, we will be forced to take action and suspend your account.

If you have any questions, please don't hesitate to ask us, we will be happy to answer them. Please feel free to contact us for further help.

Thank you,

--

Best Regards,

Quote
Topic starter Posted : 29/09/2016 1:35 am
HPFanFicArchive.Com
(@hpfanficarchive-com)
Eminent Member

I would start with adding the bots they suggested to the robots.txt including the ones shown in the logs.

Then add these mods:

Handmade "Bad behavio(u)r 2": http://efiction.org/forums/index.php?topic=8901.0

  • This one is a must.  It is going to help block a lot of the bots

Mod: Registration Notice: http://efiction.org/forums/index.php?topic=8906.0
Mod: Chapter Review Notice: http://efiction.org/forums/index.php?topic=8907.0
Mod: Block Email Domains From Registering: http://efiction.org/forums/index.php?topic=8905.0

  • These three were made by myself.  They are going to help you identify spammers and the last one can help you block them from registering.

http://www.FicSavers.Com
http://www.HPFanFicArchive.Com
http://www.FavoritesTracker.Org

ReplyQuote
Posted : 30/09/2016 2:51 am
Share: