Commit Graph

3 Commits

Author SHA1 Message Date
ghost
11aa404807 add metaYggo field index 2023-04-25 21:10:59 +03:00
ghost
df6f2a1869 implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5 2023-04-08 22:28:31 +03:00
ghost
2495a2bbc7 implement MySQL/Sphinx data model #3, add basical robots.txt support #2 2023-04-07 04:04:24 +03:00