5 Commits

Author	SHA1	Message	Date
ghost	034a683df7	add YGGstate DB crawl integration	2023-08-07 00:13:04 +03:00
ghost	d024ffd770	implement unlimited settings customization for each host	2023-08-05 19:06:39 +03:00
ghost	ab6c0379c8	implement hosts crawl queue, move robots, sitemaps, manifests to this task	2023-08-04 09:32:12 +03:00
ghost	71724ae33f	refactor manifest crawling	2023-08-04 09:00:03 +03:00
ghost	cb37c57bc4	rename example files	2023-08-03 18:49:29 +03:00