python - Very slow MongoDB count in big database -

-

i have database collection in large amount of documents (several millions). in database have (amongst others) fields _violationtype (int) , _duration (int). count amount of documents have _violationtype of 15 or less , _duration of 10 or less. execute following python script:

#!/usr/bin/env python import pymongo import timeit   client = pymongo.mongoclient('localhost', 27017) database = client['bgp_route_leaks']  collection = database['valleys']  collection.ensure_index('_violationtype', unique=false) collection.ensure_index('_duration', unique=false)  start = timeit.default_timer()  cursor = collection.find({'$and': [{'_violationtype': {'$lt': 16}}, {'_duration': {'$lt': 10}}]}, {'_duration': 1, '_id': 0})  print('explain: {}'.format(cursor.explain())) print('count: {}'.format(cursor.count())) print('time: {}'.format(timeit.default_timer() - start))

this prints out:

explain: {u'nyields': 4, u'nscannedallplans': 6244545, u'allplans': [{u'cursor': u'btreecursor _violationtype_1', u'indexbounds': {u'_violationtype': [[-1.7976931348623157e+308, 16]]}, u'nscannedobjects': 124, u'nscanned': 124, u'n': 34}, {u'cursor': u'btreecursor _duration_1', u'indexbounds': {u'_duration': [[-1.7976931348623157e+308, 10]]}, u'nscannedobjects': 6244298, u'nscanned': 6244298, u'n': 5678070}, {u'cursor': u'basiccursor', u'indexbounds': {}, u'nscannedobjects': 123, u'nscanned': 123, u'n': 36}], u'millis': 30815, u'nchunkskips': 0, u'server': u'area51:27017', u'n': 5678107, u'cursor': u'btreecursor _duration_1', u'scanandorder': false, u'indexbounds': {u'_duration': [[-1.7976931348623157e+308, 10]]}, u'nscannedobjectsallplans': 6244545, u'ismultikey': false, u'indexonly': true, u'nscanned': 6244298, u'nscannedobjects': 6244298} count: 5678107 time: 52.4030768871

while running executed db.currentop() in window, returned

{         "inprog" : [                 {                         "opid" : 15,                         "active" : true,                         "secs_running" : 4,                         "op" : "query",                         "ns" : "bgp_route_leaks.valleys",                         "query" : {                                 "$query" : {                                         "$and" : [                                                 {                                                         "_violationtype" : {                                                                 "$lt" : 16                                                         }                                                 },                                                 {                                                         "_duration" : {                                                                 "$lt" : 10                                                         }                                                 }                                         ]                                 },                                 "$explain" : true                         },                         "client" : "127.0.0.1:46819",                         "desc" : "conn1",                         "threadid" : "0x7fd69b31c700",                         "connectionid" : 1,                         "locks" : {                                 "^" : "r",                                 "^bgp_route_leaks" : "r"                         },                         "waitingforlock" : false,                         "numyields" : 5,                         "lockstats" : {                                 "timelockedmicros" : {                                         "r" : numberlong(8816104),                                         "w" : numberlong(0)                                 },                                 "timeacquiringmicros" : {                                         "r" : numberlong(4408723),                                         "w" : numberlong(0)                                 }                         }                 }         ] }

now have read common source slow mongodb queries missing indexes. however, have ensured indexes both _violationtype , _duration , explain tells me u'indexonly': true. read numa architecture slow things down , should start service via command

sudo numactl --interleave=all /usr/bin/mongod --dbpath=/var/lib/mongodb (/proc/sys/vm/zone_reclaim_mode set 0)

which know have done, still takes minute count , longer others, wondering make query faster.

running

db.runcommand({compact: 'bgp_route_leaks'})

in mongo shell has been tried no luck.

any suggestions on how counts faster?

the mongodb version 2.4.9.

if @ explain output see query using _violationtype scanning 124 objects , the query using _duration scanning 6244545 objects.

although mongodb 2.6+ can use index intersection, compound index faster.

you need create compound index on fields:

collection.create_index([("_violationtype", ascending),("_duration", ascending)]);

edit

in versions 2.4 mongodb performance improved (jira-1752).

also, it's worth noticing explain command displaying details query not count command.

unfortunately, can't use explain on count command, there ticket opened issue.

to measure performance of count command should remove explain test. also, need repeat query multiple times (100x, 1000x...) , take average correct value.


Comments

Post a Comment

Popular posts from this blog

database - VFP Grid + SQL server 2008 - grid not showing correctly -

-
i having problem, have grid called "gridbasica" , this: thisform.gridbase.recordsource = "sgviemp" sgviemp cursor made executing select sqlserver command sqlexec. when want modify registry, go form , the update there. so, when come form grid, grid isnt grid anymore, big white rectangle. reading around said: delete recordsource , asingt again. thisform.gridbase.recordsource = "" *--i sqlexe here thisform.gridbase.recordsource = "sgviemp" i , works, but, when click button modify again cursos updated modify form doesnt load fields did first time. could me out that? thanks in advance. --------------- edit ------------------ i have been doing recommended: lnselect = select(0) n = sqlexec(thisform.conexion,"select emp_ccodigo codigo, emp_cnombre 'nombre o razon social', emp_cnumrif 'r.i.f' sgviempr","sgviemp1") set echo on suspend select sgviemp zap in sgviemp *-- browse *--zaps correct
Read more

jquery - Set jPicker field to empty value -

-
im using jpicker color selector in form. have clear value set window/icon 'no color selected' state. i've tried following: $(':input[name="color"]').val(''); //access value directly. (does not affect jpicker) 2: $.jpicker.list[0].color.active.val('hex', '', this); //set color black , input value 000000, instead of empty string 3: $.jpicker.list[0].color.active.val(null, '', this); //does nothing none of 3 methods seems behave properly. in advance. this works me :) $.jpicker.list[0].color.active.val('v', null); here example: http://jsfiddle.net/3up7n/
Read more

.htaccess - htaccess convert request to clean url and add slash at the end of the url -

-
ok , searched more 4 days , more 20 posts here in stach or in websites without kind of success first here htaccess file rewriteengine on rewriterule ^([a-za-z0-9_]+)/?$ index.php?page=$1 [nc] rewriterule ^([a-za-z0-9_]+)\/([a-za-z0-9_]+)\/?$ index.php?page=$1&action=$2 [nc] i tried many codes 1 worked me can call request , works me www.example.com/pagename => www.example.com/index.php?page=pagename here questions , please provide me information "in detail" understand code well 1- how convert request automatically www.example.com/index.php?page=pagename www.example.com/pagename 2- if tried add slash @ end shows me page without images or styling mean www.example.com/pagename => works www.example.com/pagename/ => doesn't work that's :) for first part need 301 rule this: rewritecond %{the_request} /index\.php\?page=([^\s&]+) [nc] rewriterule ^ /%1? [r=301,l] for second part use absolute path in css, js, images files r
Read more