Robots

Discussion in 'Accepted' started by Oldie, Oct 6, 2014.

  1. Oldie
    Offline

    Oldie Well-Known Member

    Joined:
    Jun 6, 2014
    Messages:
    862
    Likes Received:
    259
    Location:
    PureNoob
    Country Flag:
    IGN:
    OldRanger
    Level:
    255
    Guild:
    Noobs
    Hey Staff,

    Just wondering why you let the entire forum get indexed by search engines when the in-forum search engine is already functional enough to provide search services.

    I think that the main page and announcements should be more than enough to have search engines index. Otherwise, why distribute cache copies of all our posts here across the internet? Additionally, why have constant network load from web crawlers on the website? (Sometimes there are 15+ sessions)

    Another thing, I really think that unregistered users and web crawlers should not have access to the Current Visitors page as not only does it display who's online, but what page they viewed last. It can potentially be a privacy concern for those that do not have showing their recent activity disabled.

    http://royals.ms/forum/online/?type=robot

    -Oldie
     
  2. Kimberly
    Offline

    Kimberly Well-Known Member

    Joined:
    Aug 27, 2014
    Messages:
    170
    Likes Received:
    1,138
    Country Flag:
    Search bots is good - it make sure searching "mapleroyals" ends up in high results, or even "maplestory v62 private server"

    EDIT: Searching "Maplestory v62 private server" actually shows Royals as first, so yeah.
     
    Dave Deviluke and John like this.
  3. Chokladkakan
    Offline

    Chokladkakan Web Developer

    Joined:
    Feb 2, 2014
    Messages:
    421
    Likes Received:
    986
    Assuming they have both enabled in their privacy settings, yes; since our update to the latest version of XenForo these are separate settings.

    I agree with the latter point in any case and have disabled these things for unregistered guests.

    As to the issue of crawlers: this isn't my decision, I would have to defer to someone else. I wouldn't object to a much stricter robots.txt personally. Having said that, the load from our webserver is negligible compared to the game server; the robots don't make much of a difference.
     
    Oldie likes this.
  4. Oldie
    Offline

    Oldie Well-Known Member

    Joined:
    Jun 6, 2014
    Messages:
    862
    Likes Received:
    259
    Location:
    PureNoob
    Country Flag:
    IGN:
    OldRanger
    Level:
    255
    Guild:
    Noobs
    Indexing the forums does not seem to help in this case. There are no forum links.

    yandex + bing
    [​IMG]
    [​IMG]

    google
    [​IMG]
    [​IMG]
     
  5. Matt
    Offline

    Matt Administrator

    Joined:
    May 8, 2013
    Messages:
    15,446
    Likes Received:
    19,568
    Gender:
    Male
    Location:
    United Kingdom
    Country Flag:
    IGN:
    Matt
    Level:
    N/A
    Guild:
    Staff
    You may be surprised at the amount of traffic the forum gets when people search for general MapleStory information:
    [​IMG]
    [​IMG]
    [​IMG]

    etc...

    Sf you want forum links to show up in search results then you need to be searching for something which has been posted on the forum. For example:
    "maplestory htpq quest":
    [​IMG]

    or "maplestory v62 guide":
    [​IMG]
     
  6. Marty
    Offline

    Marty Donator

    Joined:
    Dec 7, 2013
    Messages:
    3,444
    Likes Received:
    2,488
    Gender:
    Male
    Country Flag:
    IGN:
    Linyah
    Level:
    170
    [​IMG]

    Oh, do I remember this. I laughed. A lot.
     
    Katsuruka and Rob like this.
  7. Oldie
    Offline

    Oldie Well-Known Member

    Joined:
    Jun 6, 2014
    Messages:
    862
    Likes Received:
    259
    Location:
    PureNoob
    Country Flag:
    IGN:
    OldRanger
    Level:
    255
    Guild:
    Noobs
    I see. : )
    That's all google analytics yes?
     
  8. Sila
    Offline

    Sila Donator

    Joined:
    Jun 4, 2014
    Messages:
    6,199
    Likes Received:
    5,987
    Country Flag:
    IGN:
    Silachan
    Level:
    200
    Guild:
    Oblivion
    (Lol @ spongebob porn and maplestory hentai though.....)
     
    Daan likes this.
  9. Oldie
    Offline

    Oldie Well-Known Member

    Joined:
    Jun 6, 2014
    Messages:
    862
    Likes Received:
    259
    Location:
    PureNoob
    Country Flag:
    IGN:
    OldRanger
    Level:
    255
    Guild:
    Noobs
    It's only google that shows mapleroyals links so often. it really is to your advantage then, as long as google continues to show links to mapleroyals and being a top search provider.

    Bing and Yandex however, don't display links to the forums. Perhaps not at all or at much smaller amounts.
     

Share This Page