YGGverse/Yo
1
0
Fork 0
mirror of https://github.com/YGGverse/Yo.git synced 2025-01-30 08:24:16 +00:00
Code Issues Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
ghost 2023-11-24 18:00:15 +02:00
parent 6068bcd985
commit 0b60445fea
1 changed files with 17 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
README.md
View File

@ -1,29 +1,33 @@
# Yo! # Yo! Micro Web Crawler in PHP & Manticore
Micro Web Crawler in PHP & Manticore
Yo! is the next generation of [YGGo!](https://github.com/YGGverse/YGGo) project with goal to reduce server requirements and make deployment process simpler. Yo! is the next generation of [YGGo!](https://github.com/YGGverse/YGGo) project with goal to reduce server requirements and make deployment process simpler.
Index model changed to the distributed clustering model, and oriented to aggregate search results from different instances trough API.
## Implementation
Engine written in PHP and uses [Manticore](https://github.com/manticoresoftware) search engine on backend. Engine written in PHP and uses [Manticore](https://github.com/manticoresoftware) search engine on backend.
Default build adapted for [Yggdrasil](https://github.com/yggdrasil-network) eco-system but could be used to make own search project. Default build inspired and adapted for [Yggdrasil](https://github.com/yggdrasil-network) eco-system but could be used to make own search project.
Project contain: ## Components
* CLI tools for index operations * CLI tools for index operations
* JS-less frontend to make search web portal * JS-less frontend to make search web portal
* API tools to make search index distributed * API tools to make search index distributed
Features: ### Features
* MIME-based crawler with flexible filter settings * MIME-based crawler with flexible filter settings
* Page snap history with local and remote mirrors support * Page snap history with local and remote mirrors support
## CLI ### Documentation
### Index #### CLI
#### Init ##### Index
###### Init
Create initial index Create initial index
@ -32,22 +36,22 @@ php src/cli/index/init.php [reset]
``` ```
* `reset` - optional, reset existing index * `reset` - optional, reset existing index
### Document ##### Document
#### Add ###### Add
``` ```
php src/cli/document/add.php URL php src/cli/document/add.php URL
``` ```
* `URL` - add new URL to the crawl queue * `URL` - add new URL to the crawl queue
#### Crawl ###### Crawl
``` ```
php src/cli/document/crawl.php php src/cli/document/crawl.php
``` ```
#### Search ###### Search
``` ```
php src/cli/document/search.php '@title "*"' [limit] php src/cli/document/search.php '@title "*"' [limit]