trjhtr

I've written a script in python scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach the target page to scrape the required info. The below scraper first starts from the main page then track each links within class art-indexhmenu then going to one layer deep it again tracks the links within class niveau2 then again follow the links within class niveau3 then tracking the links within class art-indexbutton-wrapper it reaches the target page. Then it scrapes "model", "country" and "years" of each products. My scraper is doing it's job errorlessly. However, although it is working nice, the way I've created this scraper is very repetitive to look at. As there are always room for improvement, I suppose there should be any way to make it more robust by getting rid of banality. Thanks in advance.

This is the spider (website included):

import scrapy
from scrapy.spiders import CrawlSpider
from scrapy.http.request import Request
from scrapy.crawler import CrawlerProcess

class BikePartsSpider(scrapy.Spider):
 name = 'honda'

 def start_requests(self):
 yield Request(url = "https://www.bike-parts-honda.com/", callback = self.parse_links)

 def parse_links(self, response):
 for link in response.css('.art-indexhmenu a::attr(href)').extract():
 yield response.follow(link, callback = self.parse_inner_links) #going to one layer deep from landing page

 def parse_inner_links(self, response):
 for link in response.css('.niveau2 .art-indexbutton::attr(href)').extract():
 yield response.follow(link, callback = self.parse_cat_links) # digging deep to go another layer

 def parse_cat_links(self, response):
 for link in response.css('.niveau3 .art-indexbutton::attr(href)').extract():
 yield response.follow(link, callback = self.parse_target_links) ## go inside another layer

 def parse_target_links(self, response):
 for link in response.css('.art-indexbutton-wrapper .art-indexbutton::attr(href)').extract():
 yield response.follow(link, callback = self.parse_docs) # tracking links leading to the target page

 def parse_docs(self, response):
 items = [item for item in response.css('.titre_12_red::text').extract()]
 yield "categories":items #this is where the scraper parses the info

c = CrawlerProcess( #using CrawlerProcess() method to be able to run from the IDE
 'USER_AGENT': 'Mozilla/5.0', 
)
c.crawl(BikePartsSpider)
c.start()