YGGverse/YGGo
1
0
Fork 0
mirror of https://github.com/YGGverse/YGGo.git synced 2025-08-31 00:02:36 +00:00
Code Issues Projects Releases Wiki Activity
112 Commits 2 Branches 0 Tags
Commit Graph

6 Commits

Author SHA1 Message Date
ghost
f4db66d53f add new host.robotsPostfix rules 2023-04-09 02:14:13 +03:00
ghost
be7eae501b add host.status registry #1, #5 2023-04-09 00:28:51 +03:00
ghost
3c9bc1adaa add required user-agent construction #5 2023-04-09 00:02:31 +03:00
ghost
b819fda025 init yggdrasil robots.txt registry #5 2023-04-08 22:29:33 +03:00
ghost
df6f2a1869 implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5 2023-04-08 22:28:31 +03:00
ghost
2495a2bbc7 implement MySQL/Sphinx data model #3, add basical robots.txt support #2 2023-04-07 04:04:24 +03:00