86 Commits (6e03a76ed83e40ecf8f88131a3b9eb4b20acb04f)

Author SHA1 Message Date
ghost 142d496108 fix SQL syntax error 1 year ago
ghost d024ffd770 implement unlimited settings customization for each host 1 year ago
ghost ab6c0379c8 implement hosts crawl queue, move robots, sitemaps, manifests to this task 1 year ago
ghost 71724ae33f refactor manifest crawling 1 year ago
ghost b24d31f360 refactor cleaner, delegate tasks to crawler, init hostSetting table 1 year ago
ghost 02612d098b delete getFoundHostPage method, update API version 1 year ago
ghost 11e02da66d memory usage optimization, rename methods, remove memchached dependency from the model 1 year ago
ghost 1d7deffc4c update PR generation, delegate PR value from redirecting pages, update method names 1 year ago
ghost b7a48b905e update method names 1 year ago
ghost 9c0f361601 refactor snap storage 1 year ago
ghost a242d7b05a fix hostPageSnapId to hostPageId 1 year ago
ghost 0ea665f6e7 change method name 1 year ago
ghost 000b9ad8dd add FS cleaning features, lock execution on active crontab tasks, disable hostPageSnap/localhost untested constructions 1 year ago
ghost 3e3b7ee2ef optimize snaps, delete unused constructions 1 year ago
ghost 307eb03600 build host/host page URL in SQL query 1 year ago
ghost 8a2a79b65c add hostPageSnapStorage table to the optimization queue 1 year ago
ghost 712d67f6bf implement unlimited snap storage mirrors, delete megaCMD integration 1 year ago
ghost 9b52e3b7f5 delete unused constructions 1 year ago
ghost 1dd0a8ee2c make page rank procedural, optimize performance 1 year ago
ghost 4a4394fb27 add memcached support 1 year ago
ghost 0fb2e8a78c fix active pages total 1 year ago
ghost a4b4ea324b change top rating from hosts to pages 1 year ago
ghost 2e2501b437 implement sitemap support 1 year ago
ghost 407e0d7f18 implement top page 1 year ago
ghost 3a610d5ccb add hostPageSnap truncate command 1 year ago
ghost 443eaec64e autodelete failed snaps 1 year ago
ghost 01d5356791 remove extra brackets 2 years ago
ghost 3218add372 add custom home page reindex settings 2 years ago
ghost f4bf6b9fa4 fix crawl queue message 2 years ago
ghost 29197ab904 remove testing construction 2 years ago
ghost ed240d53b0 show available snaps only 2 years ago
ghost 5346b13602 implement custom hostPageDom elements index 2 years ago
ghost dc2d971ba0 clean up banned pages extra data 2 years ago
ghost a657d31e1d fix enum data type 2 years ago
ghost f3475035c2 show page size in explorer view, hide not available data 2 years ago
ghost ab78e17ca8 add hostPage.size collection 2 years ago
ghost 7892784f5c add httpCode column to hostPageSnapDownload table 2 years ago
ghost 0af5d165d3 remove logCrawler column not in use 2 years ago
ghost f49076bb0c index homepages and shorter URL with higher priority 2 years ago
ghost 81f7ea1e1e implement multi-storage snap downloads 2 years ago
ghost 1969707eeb integrate optional MEGA/cmd snap storage 2 years ago
ghost 50c9066f62 add tables optimization to the cron/cleaner task 2 years ago
ghost 0d19004e86 make local snap storage optimization 2 years ago
ghost 2f7d99079d implement local snaps 2 years ago
ghost d98b8f5c94 remove `hostPageToHostPage`.`quantity` field because of implements wrong duplicates counting on reindex 2 years ago
ghost eeeb3dceac implement index explorer 2 years ago
ghost 377b519a2c implement host page info mode 2 years ago
ghost 371670fadf add media referrers info 2 years ago
ghost 307ebcf0b1 add page description on title | description | keywords not empty, remove deprecated constructions 2 years ago
ghost 0fed16621a fix mime content type update 2 years ago