YGGverse
/
Yo
mirror of https://github.com/YGGverse/Yo.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Micro Web Crawler in PHP & Manticore
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
6 Commits
2 Branches
0 Tags
2.0 MiB
 
Tree: 0b60445fea
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '0b60445fea'
${ noResults }
ghost 0b60445fea
update readme 		1 year ago
src add crawl queue delay support 1 year ago
.gitignore initial commit 1 year ago
LICENSE Initial commit 1 year ago
README.md update readme 1 year ago
composer.json initial commit 1 year ago

README.md

Yo! Micro Web Crawler in PHP & Manticore

Yo! is the next generation of YGGo! project with goal to reduce server requirements and make deployment process simpler.

Index model changed to the distributed clustering model, and oriented to aggregate search results from different instances trough API.

Implementation

Engine written in PHP and uses Manticore search engine on backend.

Default build inspired and adapted for Yggdrasil eco-system but could be used to make own search project.

Components

  • CLI tools for index operations
  • JS-less frontend to make search web portal
  • API tools to make search index distributed

Features

  • MIME-based crawler with flexible filter settings
  • Page snap history with local and remote mirrors support

Documentation

CLI

Index
Init

Create initial index

php src/cli/index/init.php [reset]
  • reset - optional, reset existing index
Document
Add
php src/cli/document/add.php URL
  • URL - add new URL to the crawl queue
Crawl
php src/cli/document/crawl.php
Search
php src/cli/document/search.php '@title "*"' [limit]
  • query - required
  • limit - optional search results limit