python - Reading a tar compressed file in pandas? -


this code seems work , takes list of files , compresses them in format pandas can read, , combines them 1 location.

edit - modified code add new files (based on file not existing in tar). 

os.chdir(r'c:\\users\documents\ftp\\') saveloc = r'\\fnp\mydownloads\\' compression = "w:bz2" extension = '.tar.bz2'  filename = 'global_performance' filetype = 'performance_*.csv' tarname = saveloc+filename+extension  files = glob(filetype) tar = tarfile.open(tarname, compression) file in files:     if file not in tarname:         tar.add(file) tar.close()  filename = 'global_status' filetype = 'status_*.csv' tarname = saveloc+filename+extension  files = glob(filetype) tar = tarfile.open(tarname, compression) file in files:     if file not in tarname:         tar.add(file) tar.close()
  1. is there way pandas read tar file? can specify file know exists within file, or perhaps concat of files 1 read?
  2. being able add new files nice, assume computer has read file names determine if exists or not. there way modify code add latest files based on creation date or something? can sped compress , read newest files or perhaps within time range (30 days maybe instead of reading files in directory goes 2010)?
  3. as can see above, reading each file type within directory (based on filename) , adding separate tar. there way optimize bit instead of pasting same code on , on (there 10+ files to)?

edit - code seems operate slowly. intention find newest files not within tar , compress them , add them existing tar. based on time taking, thinking still compressing files , replacing them. can me make more efficient process.


-

Comments

Post a Comment

Popular posts from this blog

OpenCV OpenCL: Convert Mat to Bitmap in JNI Layer for Android -

there several posts converting mat bitmap using utils.mattobitmap() function. i'm assuming function can called in java layer after importing utils class. i want transfer data memory address pointed uint32_t* bmpcontent; in code below. jniexport void jnicall java_com_nod_nodcv_nodcvactivity_runfilter( jnienv *env, jclass clazz, jobject outbmp, jbytearray indata, jint width, jint height, jint choice, jint filter) { int outsz = width*height; int insz = outsz + outsz/2; androidbitmapinfo bmpinfo; if (androidbitmap_getinfo(env, outbmp, &bmpinfo) < 0) { throwjavaexception(env,"gaussianblur","error retrieving bitmap meta data"); return; } if (bmpinfo.format != android_bitmap_format_rgba_8888) { throwjavaexception(env,"gaussianblur","expecting rgba_8888 format"); return; } uint32_t* bmpcontent; if (androidbitmap_lockpixels(env, outbmp,(void**)&bmpcontent)...
Read more

android - org.xmlpull.v1.XmlPullParserException: expected: START_TAG {http://schemas.xmlsoap.org/soap/envelope/}Envelope -

i trying hit soap api in application login purpose , getting above mentioned error , can 1 tell me going wrong , first time soap apis. here code hit soap :- vector<object> args = new vector<object>(); hashtable<string, object> hashtable = new hashtable<string, object>(); hashtable.put("email", "mss.siddhart28@gmail.com"); hashtable.put("password", "siddharth"); args.add(hashtable); soapserializationenvelope envelope = new soapserializationenvelope( soapenvelope.ver11); envelope.dotnet = true; envelope.xsd = soapserializationenvelope.xsd; envelope.enc = soapserializationenvelope.enc; (new marshalhashtable()).register(envelope); httptransportse androidhttptransport = new httptransportse( "http://tempuri.org/", 15000); androidhttptransport.debug = true; string result = ""; try { soapobject request = new soapobject("http:...
Read more

python - How to remove the Xframe Options header in django? -

i have made page has iframe . inside iframe want show multiple different links article facebook, or news, or youtube video or other possible url. but, due xframe header, unable so. referred following link: https://docs.djangoproject.com/en/1.8/ref/clickjacking/ , django xframeoptionsmiddleware (x-frame-options) - allow iframe client ip but didn't help. my settings.py file's middleware_classes is: middleware_classes = ( 'django.contrib.sessions.middleware.sessionmiddleware', 'django.middleware.common.commonmiddleware', 'django.middleware.csrf.csrfviewmiddleware', 'django.contrib.auth.middleware.authenticationmiddleware', 'django.contrib.auth.middleware.sessionauthenticationmiddleware', 'django.contrib.messages.middleware.messagemiddleware', 'django.middleware.clickjacking.xframeoptionsmiddleware', ) from http://django-secure.readthedocs.org/en/latest/middleware.html , found using ...
Read more