YGGverse
/
Yo
mirror of https://github.com/YGGverse/Yo.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Micro Web Crawler in PHP & Manticore
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
7 Commits
2 Branches
0 Tags
2.0 MiB
 
Tree: 8c2bd7b55f
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '8c2bd7b55f'
${ noResults }
Yo/README.md

1.3 KiB
Raw Blame History

Yo! Micro Web Crawler in PHP & Manticore

Yo! is the next generation of YGGo! project with goal to reduce server requirements and make deployment process simpler.

Index model changed to the distributed cluster model, and oriented to aggregate search results from different instances trough API.

Codebase following minimalism such as possible.

Implementation

Engine written in PHP and uses Manticore on backend.

Default build inspired and adapted for Yggdrasil eco-system but could be used to make own search project.

Components

  • CLI tools for index operations
  • JS-less frontend to make search web portal
  • API tools to make search index distributed

Features

  • MIME-based crawler with flexible filter settings
  • Page snap history with local and remote mirrors support

Documentation

CLI

Index
Init

Create initial index

php src/cli/index/init.php [reset]
  • reset - optional, reset existing index
Document
Add
php src/cli/document/add.php URL
  • URL - add new URL to the crawl queue
Crawl
php src/cli/document/crawl.php
Search
php src/cli/document/search.php '@title "*"' [limit]
  • query - required
  • limit - optional search results limit