python - PySpark broadcast value to dictionary -


have pyspark broadcast value content this:

[('b000jz4hqo', {'rom': 2.4051362683438153, 'clickart': 56.65432098765432, '950': 254.94444444444443, 'image': 3.6948470209339774, 'premier': 9.27070707070707, '000': 6.218157181571815, 'dvd': 1.287598204264871, 'broderbund': 22.169082125603865, 'pack': 2.98180636777128}), ('b0006zf55o', {'laptops': 11.588383838383837, 'desktops': 12.74722222222222, 'backup': 2.8015873015873014, 'win': 0.501859142607174, 'ca': 9.10515873015873, 'v11': 50.98888888888888, '30u': 84.98148148148148, '30pk': 254.94444444444443, 'desktop': 2.23635477582846, '1': 0.3231235037318687, 'arcserve': 24.28042328042328, 'computer': 0.6965695203400122, 'lap': 127.47222222222221, 'oem': 46.35353535353535, 'international': 9.44238683127572, 'associates': 7.284126984126985})]

so key->list broadcast variable.

attempts convert broadcast.value dictionary results in

typeerror: unhashable type: 'dict'

using code like

from itertools import izip amazonweightsbroadcast = sc.broadcast(amazonweightsrdd.collect()) = iter(amazonweightsbroadcast.value) amazonweightsdict = dict(izip(i, i))

also tried (gives same "unshapable" error):

amazonweightsdict = dict(amazonweightsbroadcast.value[i:i+2] in range(0, len(amazonweightsbroadcast.value), 2))

so if it's not possible convert broadcast variable dictionary, better solution lookup value-list key?

python 2.7.6 spark 1.3.1

took me while.. problem in how broadcast variable created. had use .collectasmap() , not .collect() working expected.


-

Comments

Post a Comment

Popular posts from this blog

OpenCV OpenCL: Convert Mat to Bitmap in JNI Layer for Android -

there several posts converting mat bitmap using utils.mattobitmap() function. i'm assuming function can called in java layer after importing utils class. i want transfer data memory address pointed uint32_t* bmpcontent; in code below. jniexport void jnicall java_com_nod_nodcv_nodcvactivity_runfilter( jnienv *env, jclass clazz, jobject outbmp, jbytearray indata, jint width, jint height, jint choice, jint filter) { int outsz = width*height; int insz = outsz + outsz/2; androidbitmapinfo bmpinfo; if (androidbitmap_getinfo(env, outbmp, &bmpinfo) < 0) { throwjavaexception(env,"gaussianblur","error retrieving bitmap meta data"); return; } if (bmpinfo.format != android_bitmap_format_rgba_8888) { throwjavaexception(env,"gaussianblur","expecting rgba_8888 format"); return; } uint32_t* bmpcontent; if (androidbitmap_lockpixels(env, outbmp,(void**)&bmpcontent)...
Read more

android - org.xmlpull.v1.XmlPullParserException: expected: START_TAG {http://schemas.xmlsoap.org/soap/envelope/}Envelope -

i trying hit soap api in application login purpose , getting above mentioned error , can 1 tell me going wrong , first time soap apis. here code hit soap :- vector<object> args = new vector<object>(); hashtable<string, object> hashtable = new hashtable<string, object>(); hashtable.put("email", "mss.siddhart28@gmail.com"); hashtable.put("password", "siddharth"); args.add(hashtable); soapserializationenvelope envelope = new soapserializationenvelope( soapenvelope.ver11); envelope.dotnet = true; envelope.xsd = soapserializationenvelope.xsd; envelope.enc = soapserializationenvelope.enc; (new marshalhashtable()).register(envelope); httptransportse androidhttptransport = new httptransportse( "http://tempuri.org/", 15000); androidhttptransport.debug = true; string result = ""; try { soapobject request = new soapobject("http:...
Read more

python - How to remove the Xframe Options header in django? -

i have made page has iframe . inside iframe want show multiple different links article facebook, or news, or youtube video or other possible url. but, due xframe header, unable so. referred following link: https://docs.djangoproject.com/en/1.8/ref/clickjacking/ , django xframeoptionsmiddleware (x-frame-options) - allow iframe client ip but didn't help. my settings.py file's middleware_classes is: middleware_classes = ( 'django.contrib.sessions.middleware.sessionmiddleware', 'django.middleware.common.commonmiddleware', 'django.middleware.csrf.csrfviewmiddleware', 'django.contrib.auth.middleware.authenticationmiddleware', 'django.contrib.auth.middleware.sessionauthenticationmiddleware', 'django.contrib.messages.middleware.messagemiddleware', 'django.middleware.clickjacking.xframeoptionsmiddleware', ) from http://django-secure.readthedocs.org/en/latest/middleware.html , found using ...
Read more