18 Commits

Author	SHA1	Message	Date
ghost	2c5ca1b630	fix image description duplicate	2023-05-09 15:23:32 +03:00
ghost	1c7cca1446	fix UNIQUE index relation	2023-05-09 14:10:08 +03:00
ghost	28bf526d53	add host nsfw settings	2023-05-09 13:26:19 +03:00
ghost	23ead4e12c	update page / image description models, implement history snap crawling	2023-05-09 08:19:49 +03:00
ghost	0e9d29675f	implement host page description history crawling	2023-05-09 01:29:32 +03:00
ghost	25b6bce2ec	add crawler/cleaner logs	2023-05-08 11:04:59 +03:00
ghost	b6605b9132	implement not reachable resources ban feature with timeout to prevent extra http requests	2023-05-06 08:45:37 +03:00
ghost	702a14b634	add mime content type crawling #1	2023-05-06 07:25:54 +03:00
ghost	5999fb3a73	add distributed hosts crawling using yggo nodes manifest	2023-05-05 05:26:53 +03:00
ghost	d4f66c83e7	fix image crawling errors	2023-05-04 08:51:45 +03:00
ghost	68581960a3	add image.data field	2023-05-04 05:19:29 +03:00
ghost	0741a3e9ef	implement image crawler	2023-05-04 01:04:39 +03:00
ghost	78931ebc74	normalize host image description storage	2023-05-03 21:52:00 +03:00
ghost	db617f9939	refactor image storage model	2023-05-03 21:27:15 +03:00
ghost	6d8f4f4882	create manifests registry	2023-05-03 09:22:14 +03:00
ghost	11aa404807	add metaYggo field index	2023-04-25 21:10:59 +03:00
ghost	df6f2a1869	implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5	2023-04-08 22:28:31 +03:00
ghost	2495a2bbc7	implement MySQL/Sphinx data model #3, add basical robots.txt support #2	2023-04-07 04:04:24 +03:00