13 Commits

Author	SHA1	Message	Date
ghost	25b6bce2ec	add crawler/cleaner logs	2023-05-08 11:04:59 +03:00
ghost	b6605b9132	implement not reachable resources ban feature with timeout to prevent extra http requests	2023-05-06 08:45:37 +03:00
ghost	702a14b634	add mime content type crawling #1	2023-05-06 07:25:54 +03:00
ghost	5999fb3a73	add distributed hosts crawling using yggo nodes manifest	2023-05-05 05:26:53 +03:00
ghost	d4f66c83e7	fix image crawling errors	2023-05-04 08:51:45 +03:00
ghost	68581960a3	add image.data field	2023-05-04 05:19:29 +03:00
ghost	0741a3e9ef	implement image crawler	2023-05-04 01:04:39 +03:00
ghost	78931ebc74	normalize host image description storage	2023-05-03 21:52:00 +03:00
ghost	db617f9939	refactor image storage model	2023-05-03 21:27:15 +03:00
ghost	6d8f4f4882	create manifests registry	2023-05-03 09:22:14 +03:00
ghost	11aa404807	add metaYggo field index	2023-04-25 21:10:59 +03:00
ghost	df6f2a1869	implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5	2023-04-08 22:28:31 +03:00
ghost	2495a2bbc7	implement MySQL/Sphinx data model #3, add basical robots.txt support #2	2023-04-07 04:04:24 +03:00