See below my method for scraping groups including comments and replies using the latest master. Scraping 100 posts including comments and replies takes ~ 10 minutes with this configuration. After 100 posts I am getting TemporarilyBanned
though. Then when it attempts to start again where it left of after the 10 minutes wait it will get TemporarilyBanned
right again [tested over 30 minutes].
Any suggestions? I feel the delays are very high already.
def scrape_group_posts[self, group_ids: List[Union[int, str]]]: def handle_pagination_url[url]: nonlocal start_url start_url = url for k, group_id in enumerate[group_ids, 1]: group_name = self.group_information[group_id]['name'] log.info[f"[{k}] Scraping group: {group_name}..."] start_url = None post_counter = 0 keep_alive = True while keep_alive: try: posts = self.get_group_posts[ group=group_id, options={ "comments": "generator" if config.SCRAPE_COMMENTS else False, "comment_start_url": start_url, "comment_request_url_callback": handle_pagination_url } ] while post := next[posts, None]: post_counter += 1 if post["time"]