noindex - Controlling Search Engine Index Removals -


my site has particular pages are:

  1. already indexed in search engines, want remove them indexes.
  2. numerous, dynamic (based on query string).
  3. a bit "heavy." (an overzealous bot can strain server more i'd like.)

because of #2, i'm going let them removed naturally, need settle on plan.

i started out doing following:

  1. bots: abort execution using user-agent detection in application, , send blank response. (i don't mind if bots slip through , render real page, i'm blocking common ones.)
  2. bots: throw 403 (forbidden) response code.
  3. all clients: send "x-robots-tag: noindex" header.
  4. all clients: added rel="nofollow" links lead these pages.
  5. did not disallow bots pages in robots.txt. (i think it's useful disallow bots if beginning, or else after pages completely removed search engines; otherwise, engines can't crawl/access pages discover/honor noindex header, wouldn't remove them. mention because think robots.txt might commonly misunderstood, , might suggested inappropriate silver bullet.)

however, since then, think of steps either useless toward goal, or problematic.

  • i'm not sure if throwing 403 bots idea. search engines see , disregard x-robots-tag? better let them respond 200?
  • i think rel="nofollow" potentially affects target page rank, , doesn't affect crawling @ all.

the rest of plan seems okay (correct me if i'm wrong), i'm not sure above bullets in grand scheme.

i think plan:

  1. bots: abort execution using user-agent detection in application, , send blank response. (i don't mind if bots slip through , render real page, i'm blocking common ones.)
  2. bots: send 410 (gone) response code.
    "in general, webmasters little caught in tiny little details , if page gone, it's fine serve 404, if know it's gone real it's fine serve 410,"
    - http://goo.gl/awjdez
  3. all clients: send "x-robots-tag: noindex" header. think extraneous known bots got 410, cover unknown engines' bots.
  4. all clients: add rel="nofollow" links lead these pages. isn't necessary, wouldn't hurt.
  5. do not disallow bots pages in robots.txt. (it's useful disallow bots if beginning, or else after pages completely removed search engines; otherwise, engines can't crawl/access pages discover/honor noindex header, wouldn't remove them. mention because think robots.txt might commonly misunderstood, , might suggested inappropriate silver bullet.)

-

Comments

  1. seravina danniella15 November 2017 at 01:55

    This article is very informative and easy to understand. Thank you for sharing!


    Web Designer

    ReplyDelete

Post a Comment

Popular posts from this blog

OpenCV OpenCL: Convert Mat to Bitmap in JNI Layer for Android -

there several posts converting mat bitmap using utils.mattobitmap() function. i'm assuming function can called in java layer after importing utils class. i want transfer data memory address pointed uint32_t* bmpcontent; in code below. jniexport void jnicall java_com_nod_nodcv_nodcvactivity_runfilter( jnienv *env, jclass clazz, jobject outbmp, jbytearray indata, jint width, jint height, jint choice, jint filter) { int outsz = width*height; int insz = outsz + outsz/2; androidbitmapinfo bmpinfo; if (androidbitmap_getinfo(env, outbmp, &bmpinfo) < 0) { throwjavaexception(env,"gaussianblur","error retrieving bitmap meta data"); return; } if (bmpinfo.format != android_bitmap_format_rgba_8888) { throwjavaexception(env,"gaussianblur","expecting rgba_8888 format"); return; } uint32_t* bmpcontent; if (androidbitmap_lockpixels(env, outbmp,(void**)&bmpcontent)...
Read more

android - org.xmlpull.v1.XmlPullParserException: expected: START_TAG {http://schemas.xmlsoap.org/soap/envelope/}Envelope -

i trying hit soap api in application login purpose , getting above mentioned error , can 1 tell me going wrong , first time soap apis. here code hit soap :- vector<object> args = new vector<object>(); hashtable<string, object> hashtable = new hashtable<string, object>(); hashtable.put("email", "mss.siddhart28@gmail.com"); hashtable.put("password", "siddharth"); args.add(hashtable); soapserializationenvelope envelope = new soapserializationenvelope( soapenvelope.ver11); envelope.dotnet = true; envelope.xsd = soapserializationenvelope.xsd; envelope.enc = soapserializationenvelope.enc; (new marshalhashtable()).register(envelope); httptransportse androidhttptransport = new httptransportse( "http://tempuri.org/", 15000); androidhttptransport.debug = true; string result = ""; try { soapobject request = new soapobject("http:...
Read more

python - How to remove the Xframe Options header in django? -

i have made page has iframe . inside iframe want show multiple different links article facebook, or news, or youtube video or other possible url. but, due xframe header, unable so. referred following link: https://docs.djangoproject.com/en/1.8/ref/clickjacking/ , django xframeoptionsmiddleware (x-frame-options) - allow iframe client ip but didn't help. my settings.py file's middleware_classes is: middleware_classes = ( 'django.contrib.sessions.middleware.sessionmiddleware', 'django.middleware.common.commonmiddleware', 'django.middleware.csrf.csrfviewmiddleware', 'django.contrib.auth.middleware.authenticationmiddleware', 'django.contrib.auth.middleware.sessionauthenticationmiddleware', 'django.contrib.messages.middleware.messagemiddleware', 'django.middleware.clickjacking.xframeoptionsmiddleware', ) from http://django-secure.readthedocs.org/en/latest/middleware.html , found using ...
Read more