67 Commits (4a4394fb270dcea544ec84aca88ebfe02b1603bb)

Author SHA1 Message Date
ghost 4a4394fb27 add memcached support 1 year ago
ghost 0fb2e8a78c fix active pages total 1 year ago
ghost a4b4ea324b change top rating from hosts to pages 1 year ago
ghost 2e2501b437 implement sitemap support 1 year ago
ghost 407e0d7f18 implement top page 1 year ago
ghost 3a610d5ccb add hostPageSnap truncate command 1 year ago
ghost 443eaec64e autodelete failed snaps 1 year ago
ghost 01d5356791 remove extra brackets 2 years ago
ghost 3218add372 add custom home page reindex settings 2 years ago
ghost f4bf6b9fa4 fix crawl queue message 2 years ago
ghost 29197ab904 remove testing construction 2 years ago
ghost ed240d53b0 show available snaps only 2 years ago
ghost 5346b13602 implement custom hostPageDom elements index 2 years ago
ghost dc2d971ba0 clean up banned pages extra data 2 years ago
ghost a657d31e1d fix enum data type 2 years ago
ghost f3475035c2 show page size in explorer view, hide not available data 2 years ago
ghost ab78e17ca8 add hostPage.size collection 2 years ago
ghost 7892784f5c add httpCode column to hostPageSnapDownload table 2 years ago
ghost 0af5d165d3 remove logCrawler column not in use 2 years ago
ghost f49076bb0c index homepages and shorter URL with higher priority 2 years ago
ghost 81f7ea1e1e implement multi-storage snap downloads 2 years ago
ghost 1969707eeb integrate optional MEGA/cmd snap storage 2 years ago
ghost 50c9066f62 add tables optimization to the cron/cleaner task 2 years ago
ghost 0d19004e86 make local snap storage optimization 2 years ago
ghost 2f7d99079d implement local snaps 2 years ago
ghost d98b8f5c94 remove `hostPageToHostPage`.`quantity` field because of implements wrong duplicates counting on reindex 2 years ago
ghost eeeb3dceac implement index explorer 2 years ago
ghost 377b519a2c implement host page info mode 2 years ago
ghost 371670fadf add media referrers info 2 years ago
ghost 307ebcf0b1 add page description on title | description | keywords not empty, remove deprecated constructions 2 years ago
ghost 0fed16621a fix mime content type update 2 years ago
ghost db0e66c846 refactor to mime-based content index #1 2 years ago
ghost 0ffcee1efb fix image description updates timing 2 years ago
ghost 2c5ca1b630 fix image description duplicate 2 years ago
ghost 28bf526d53 add host nsfw settings 2 years ago
ghost dfca5570c6 remove unused construction 2 years ago
ghost ef4de6b245 fix image search page errors 2 years ago
ghost 23ead4e12c update page / image description models, implement history snap crawling 2 years ago
ghost 0e9d29675f implement host page description history crawling 2 years ago
ghost 32d0f390d3 update http code and mime type on page/image ban event 2 years ago
ghost 8fbd7f3516 count totals using sphinx index instead of database 2 years ago
ghost 25b6bce2ec add crawler/cleaner logs 2 years ago
ghost 6c41dd5831 fix ban time update / count affected rows only 2 years ago
ghost b6605b9132 implement not reachable resources ban feature with timeout to prevent extra http requests 2 years ago
ghost 702a14b634 add mime content type crawling #1 2 years ago
ghost 5999fb3a73 add distributed hosts crawling using yggo nodes manifest 2 years ago
ghost f0b2eb1613 show images total instead of pages in placeholder on image search page 2 years ago
ghost 297563d4a5 display related pages in priority to the unique host by rank, rand() order 2 years ago
ghost 34b7291228 add related to image hostpages limit 2 years ago
ghost adc791f378 fix updateTime init 2 years ago