Web scraping with scrapy -

- 

from scrapy.spider import basespider scrapy.selector import selector scrapy.exceptions import closespider scrapy.http import request botg.items import botgitem   url = "http://store.tcgplayer.com/magic/born-of-the-gods?pagenumber=%d"  class myspider(basespider): name = "tcg" allowed_domains = ["tcgplayer.com"] start_urls = [url % 1]  def __init__(self):     self.page_number = 1  def parse(self, response):     print self.page_number     print "--------------------break-------------------------"      sel = selector(response)     titles = sel.xpath("//div[@class='magiccard']")     if not titles:         raise closespider('no more pages')      title in titles:         item = botgitem()         item["cardname"] = title.xpath(".//li[@class='cardname']/a/text()").extract()[0]         item["rarity"] = title.xpath(".//li[@href='/magic/born-of-the-gods']/text()").extract()          vendor = title.xpath(".//tr[@class='vendor ']")         item["price"] = vendor.xpath("normalize-space(.//td[@class='price']/text())").extract()         item["quantity"] = vendor.xpath("normalize-space(.//td[@class='quantity']/text())").extract()         item["shipping"] = vendor.xpath("normalize-space(.//span[@class='shippingamount']/text())").extract()         item["condition"] = vendor.xpath("normalize-space(.//td[@class='condition']/a/text())").extract()         item["vendors"] = vendor.xpath("normalize-space(.//td[@class='seller']/a/text())").extract()          yield item      self.page_number += 1     yield request(url % self.page_number)

i using code scrape page, not able "rarity" scrape. appreciated. else seems work, tell me "[0]" after .extract() in line cardname item.

for rarity field, suggest:

  • you text representation of <ul> containing <li class="cardname">,
  • extract what's after "rarity: " regex

something this:

for title in titles:     item = botgitem()     item["rarity"] = title.xpath('string(.//ul[li[@class="cardname"]])').re(r'rarity:\s*(\w+)')

about 2nd question, .extract() extracts list of strings, [0] selects 1st element of list


Comments

Post a Comment

Popular posts from this blog

database - VFP Grid + SQL server 2008 - grid not showing correctly -

-
i having problem, have grid called "gridbasica" , this: thisform.gridbase.recordsource = "sgviemp" sgviemp cursor made executing select sqlserver command sqlexec. when want modify registry, go form , the update there. so, when come form grid, grid isnt grid anymore, big white rectangle. reading around said: delete recordsource , asingt again. thisform.gridbase.recordsource = "" *--i sqlexe here thisform.gridbase.recordsource = "sgviemp" i , works, but, when click button modify again cursos updated modify form doesnt load fields did first time. could me out that? thanks in advance. --------------- edit ------------------ i have been doing recommended: lnselect = select(0) n = sqlexec(thisform.conexion,"select emp_ccodigo codigo, emp_cnombre 'nombre o razon social', emp_cnumrif 'r.i.f' sgviempr","sgviemp1") set echo on suspend select sgviemp zap in sgviemp *-- browse *--zaps correct
Read more

jquery - Set jPicker field to empty value -

-
im using jpicker color selector in form. have clear value set window/icon 'no color selected' state. i've tried following: $(':input[name="color"]').val(''); //access value directly. (does not affect jpicker) 2: $.jpicker.list[0].color.active.val('hex', '', this); //set color black , input value 000000, instead of empty string 3: $.jpicker.list[0].color.active.val(null, '', this); //does nothing none of 3 methods seems behave properly. in advance. this works me :) $.jpicker.list[0].color.active.val('v', null); here example: http://jsfiddle.net/3up7n/
Read more

.htaccess - htaccess convert request to clean url and add slash at the end of the url -

-
ok , searched more 4 days , more 20 posts here in stach or in websites without kind of success first here htaccess file rewriteengine on rewriterule ^([a-za-z0-9_]+)/?$ index.php?page=$1 [nc] rewriterule ^([a-za-z0-9_]+)\/([a-za-z0-9_]+)\/?$ index.php?page=$1&action=$2 [nc] i tried many codes 1 worked me can call request , works me www.example.com/pagename => www.example.com/index.php?page=pagename here questions , please provide me information "in detail" understand code well 1- how convert request automatically www.example.com/index.php?page=pagename www.example.com/pagename 2- if tried add slash @ end shows me page without images or styling mean www.example.com/pagename => works www.example.com/pagename/ => doesn't work that's :) for first part need 301 rule this: rewritecond %{the_request} /index\.php\?page=([^\s&]+) [nc] rewriterule ^ /%1? [r=301,l] for second part use absolute path in css, js, images files r
Read more