YGGverse
/
YGGo
mirror of https://github.com/YGGverse/YGGo.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
243 Commits
2 Branches
0 Tags
4.3 MiB
Tree: b6605b9132
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'b6605b9132'
${ noResults }
Commit Graph

7 Commits (b6605b91322508d0d054e207805052679a4238c4)

Author SHA1 Message Date
ghost b6605b9132 implement not reachable resources ban feature with timeout to prevent extra http requests
2 years ago
ghost f88d2ee9ff implement MIME content-type crawler filter
2 years ago
ghost 5999fb3a73 add distributed hosts crawling using yggo nodes manifest
2 years ago
ghost 79878d17fe add crawler / proxy user agent settings
2 years ago
ghost 0741a3e9ef implement image crawler
2 years ago
ghost eb3e70a7b7 fix robots.txt conditions
2 years ago
ghost 8e8d89db0e implement database cleaner
2 years ago