A way to limit the Zyte API requests

[BurnzZ]

I'm thinking of a feature where the user can indicate the max number of Zyte API requests for a given spider crawl. When this number is reached, the spider shuts down.

The main use case for this is to limit the costs per crawl.

[Gallaecio]

Wouldn’t CLOSESPIDER_PAGECOUNT do the job in most cases?

[BurnzZ]

Yes, that's right. But it would also include non-ZAPI requests to the count.

EDIT: I also realized that CLOSESPIDER_PAGECOUNT doesn't guarantee that the number of requests won't stop at that point. Any remaining requests in the queue/engine would still be executed.

[BurnzZ]

An alternative: https://github.com/croqaz/scrapy-count-filter

Though it still doesn't address the issue of directly limiting the number of ZAPI requests.

[BurnzZ]

Closing as #114 has been merged