13 Commits

Author	SHA1	Message	Date
ghost	6d8f4f4882	create manifests registry	2023-05-03 09:22:14 +03:00
ghost	ec20435790	remove presets registry (because provided in the node API)	2023-05-03 04:13:32 +03:00
ghost	11aa404807	add metaYggo field index	2023-04-25 21:10:59 +03:00
ghost	352466ad03	update host.robotsPostfix registry	2023-04-10 03:19:08 +03:00
ghost	6550eb310f	update host.robotsPostfix rules	2023-04-09 03:10:42 +03:00
ghost	6cee58214e	update host.robotsPostfix rules	2023-04-09 03:05:43 +03:00
ghost	6f4daf7a25	update host.robotsPostfix rule	2023-04-09 02:19:07 +03:00
ghost	f4db66d53f	add new host.robotsPostfix rules	2023-04-09 02:14:13 +03:00
ghost	be7eae501b	add host.status registry #1, #5	2023-04-09 00:28:51 +03:00
ghost	3c9bc1adaa	add required user-agent construction #5	2023-04-09 00:02:31 +03:00
ghost	b819fda025	init yggdrasil robots.txt registry #5	2023-04-08 22:29:33 +03:00
ghost	df6f2a1869	implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5	2023-04-08 22:28:31 +03:00
ghost	2495a2bbc7	implement MySQL/Sphinx data model #3, add basical robots.txt support #2	2023-04-07 04:04:24 +03:00