scrapy notes

use in shell

1. scrapy shell url

css selector:
response.css('small.author')

<small class = "author"> william<> --> html will look something like this

response.css('small.author').extract()

to get the actual html data

Now to remove the html selector:

response.css('small.author::text').extract()

To get the result as string and not as list and to select the first element

response.css('small.author::text')[0].extract()

or

response.css('small.author::text').extract_first()

for doing scrapy shell website
to load a new website while we are in shell we can use
fetch(url)

To create a new request
scrapy.Request(url)