python - Investigating different datatypes in Pandas DataFrame -

-

i have 4 files want read python / pandas, files are: https://github.com/kelsey9649/cs8370group/tree/master/tafengdataset stripped away first row (column titles in chinese) in 4 files. other that, 4 files supposed have same format.

now want read them , merge 1 big dataframe. tried using

pars = {'sep':          ';',             'header':       none,             'names':        ['date','customer_id','age','area','prod_class','prod_id','amount','asset','price'],              'parse_dates':  [0]}  df = pd.dataframe() in ('01', '02', '12', '11'):     df = df.append(pd.read_csv(cfg.abspath+'d'+i,**pars))

but: file d11 gives me different format of single columns , cannot merged properly. file contains on 200k lines , cannot problem in file mentioned above, assuming has same format, there's small difference in format.

what's easiest way of investigating problem? obviously, cannot check every single line in file...

when read 3 working files , merge them; , read d11 independetly, line

a = pd.read_csv(cfg.abspath+'d11',**pars)

still gives me following warning:

c:\python27\lib\site-packages\pandas\io\parsers.py:1130: dtypewarning: columns ( 1,4,5,6,7,8) have mixed types. specify dtype option on import or set low_memory= false.   data = self._reader.read(nrows)

using method .info() in pandas (for a , df) yields:

<class 'pandas.core.frame.dataframe'> int64index: 594119 entries, 0 178215 data columns (total 9 columns): date           594119 non-null datetime64[ns] customer_id    594119 non-null int64 age            594119 non-null object area           594119 non-null object prod_class     594119 non-null int64 prod_id        594119 non-null int64 amount         594119 non-null int64 asset          594119 non-null int64 price          594119 non-null int64 dtypes: datetime64[ns](1), int64(6), object(2)  <class 'pandas.core.frame.dataframe'> int64index: 223623 entries, 0 223622 data columns (total 9 columns): date           223623 non-null object customer_id    223623 non-null object age            223623 non-null object area           223623 non-null object prod_class     223623 non-null object prod_id        223623 non-null object amount         223623 non-null object asset          223623 non-null object price          223623 non-null object

even if use dtype-option on import, somehow still scared of wrong/bad results there might happen wrong casting of datatypes while importing!?

how overcome , solve issue? lot

whenever have problem boring done hand, solution write program: 

for col in ('age', 'area'):     i, val in enumerate(a[col]):         try:             int(val)         except:             print('line {}: {} = {}'.format(i, col, val))

this show lines in file non-integer values in age , area columns. first step in debugging problem. once know problematic values are, can better decide how deal them -- maybe pre-processing (cleaning) data file, or using pandas code select , fix problematic values.


Comments

Post a Comment

Popular posts from this blog

C# random value from dictionary and tuple -

-
i have following code, , need random item contains string "region_1" dictionary. public static dictionary<int, tuple<string, int, currencytype>> itemarray = new dictionary<int, tuple<string, int, currencytype>>() { { 0xb3e, new tuple<string, int, currencytype>("region_1", 1500, currencytype.fame) } }; public static int generateitemid(string shopid) { var generateditem = itemarray.elementat(new random().next(0, itemarray.count)).key; } how select this? it's not possible efficiently... first create static random @ class level... prevent non-random behaviour if run query on short period of time... (it's seeded discrete clock) static random rnd = new random(); then: var item = itemarray.values .where(t => t.item1 == shopid) .orderby(_ => rnd.next()) .firstordefault()
Read more

cgi - How do I interpret URLs without extension as files rather than missing directories in nginx? -

-
so i'm trying install zoneminder , have run under nginx, has few compiled files need run using fast-cgi. these files lack extension. if file has extension, there no issue , nginx interprets file , return 404 if can't find it. if has no extension, assume it's directory , return 404 when can't find sort of index page. here have now: # security camera domain. server { listen 888; server_name mydomain.com; root /srv/http/zm; # enable php support. include php.conf; location / { index index.html index.htm index.php; } # enable cgi support. location ~ ^/cgi-bin/(.+)$ { alias /srv/cgi-bin/zm/; fastcgi_pass unix:/run/fcgiwrap.sock; fastcgi_param script_filename $document_root/$1; include fastcgi.conf; } } the idea is, under cgi-bin directory goes through fastcgi pipe. any idea on howi can fix this? upon closer inspection, realized zoneminder has 2 cgi sc...
Read more

.htaccess - htaccess convert request to clean url and add slash at the end of the url -

-
ok , searched more 4 days , more 20 posts here in stach or in websites without kind of success first here htaccess file rewriteengine on rewriterule ^([a-za-z0-9_]+)/?$ index.php?page=$1 [nc] rewriterule ^([a-za-z0-9_]+)\/([a-za-z0-9_]+)\/?$ index.php?page=$1&action=$2 [nc] i tried many codes 1 worked me can call request , works me www.example.com/pagename => www.example.com/index.php?page=pagename here questions , please provide me information "in detail" understand code well 1- how convert request automatically www.example.com/index.php?page=pagename www.example.com/pagename 2- if tried add slash @ end shows me page without images or styling mean www.example.com/pagename => works www.example.com/pagename/ => doesn't work that's :) for first part need 301 rule this: rewritecond %{the_request} /index\.php\?page=([^\s&]+) [nc] rewriterule ^ /%1? [r=301,l] for second part use absolute path in css, js, images files r...
Read more