8 Commits

Author	SHA1	Message	Date
ghost	a5f5541395	skip robots:noindex page without extra actions	2023-04-29 08:58:48 +03:00
ghost	11aa404807	add metaYggo field index	2023-04-25 21:10:59 +03:00
ghost	8671fc4bde	implement page ranking	2023-04-25 16:54:01 +03:00
ghost	9916fb701f	implement basic api	2023-04-23 03:01:51 +03:00
ghost	5c8d299a4a	add meta:robots tag support #2	2023-04-09 03:28:31 +03:00
ghost	8e8d89db0e	implement database cleaner	2023-04-09 00:06:28 +03:00
ghost	df6f2a1869	implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5	2023-04-08 22:28:31 +03:00
ghost	2495a2bbc7	implement MySQL/Sphinx data model #3, add basical robots.txt support #2	2023-04-07 04:04:24 +03:00