YGGo! Distributed Web Search Engine
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

135 lines
4.4 KiB

2 years ago
### YGGo! - Open Source Web Search Engine
2 years ago
Проект присвячується захисникам міста Бахмут
2 years ago
Written by inspiration to explore [Yggdrasil](https://yggdrasil-network.github.io) ecosystem, because of last [YaCy](https://yacy.net/) node there was discontinued.
2 years ago
This engine also could be useful for crawling regular websites, small business resources, local networks.
2 years ago
2 years ago
The project goal - simple interface, clear architecture and lightweight server requirement.
2 years ago
2 years ago
#### Online instances
2 years ago
* [http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo](http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo)
2 years ago
* [http://94.140.114.241/yggo/](http://94.140.114.241/yggo)
2 years ago
2 years ago
#### Screenshots
2 years ago
![Home page](https://github.com/YGGverse/YGGo/blob/main/media/main-sm.png?raw=true)
2 years ago
https://github.com/YGGverse/YGGo/tree/main/media
2 years ago
2 years ago
#### License
2 years ago
* Engine sources [MIT License](https://github.com/YGGverse/YGGo/blob/main/LICENSE)
* Home page animation by [alvarotrigo](https://codepen.io/alvarotrigo/pen/GRvYNax)
2 years ago
2 years ago
#### Requirements
```
2 years ago
php8^
php-dom
2 years ago
php-pdo
2 years ago
php-curl
2 years ago
php-gd
php-mysql
2 years ago
sphinxsearch
2 years ago
```
#### Installation
2 years ago
* The webroot dir is `/public`
2 years ago
* Single configuration file placed here `/config/app.php.txt` and need to be configured and renamed to `/config/app.php`
2 years ago
* By the idea, script automaticaly generates database structure in `/storage` folder (where could be nice to collect other variative and tmp data - like logs, etc). Make sure storage folder writable.
2 years ago
* Set up the `/crontab/crawler.php` script for execution every the minute, but it mostly related of the configs and targetal network volume, there is no debug implemented yet, so let's silentize it by `/dev/null`
2 years ago
* Script has no MVC model, because of super simple. It's is just 2 files, and everything else stored incapsulated in `/library` classes.
2 years ago
#### Configuration
##### Crontab
```
@reboot searchd
@reboot indexer --all --rotate
0 * * * * indexer --all --rotate
0 0 * * * cd /YGGo/crontab && php cleaner.php > /dev/null 2>&1
* * * * * cd /YGGo/crontab && php crawler.php > /dev/null 2>&1
```
2 years ago
#### JSON API
2 years ago
Build third party applications / index distribution.
2 years ago
2 years ago
Could be enabled or disabled by `API_ENABLED` option
2 years ago
###### Address
```
/api.php
```
##### Search API
Returns search results.
2 years ago
2 years ago
Could be enabled or disabled by `API_SEARCH_ENABLED` option
###### Request attributes
```
2 years ago
GET action=search - required
GET query={string} - optional, search request, empty if not provided
GET page={int} - optional, search results page, 1 if not provided
```
##### Hosts distribution API
2 years ago
Returns node hosts collected with fields provided in `API_HOSTS_FIELDS` option.
2 years ago
2 years ago
Could be enabled or disabled by `API_HOSTS_ENABLED` option
###### Request attributes
```
2 years ago
GET action=hosts - required
```
2 years ago
#### Roadmap / ideas
2 years ago
* [x] Web pages full text ranking search
2 years ago
* [x] Make search results pagination
* [x] Add robots.txt support (Issue #2)
2 years ago
* [ ] Improve yggdrasil links detection, add .ygg domain zone support
2 years ago
* [ ] Make page description visible - based on the cached content dump, when website description tag not available, add condition highlights
2 years ago
* [ ] Images search (basically implemented but requires testing and some performance optimization)
* [x] Index cleaner
2 years ago
* [ ] Crawl queue balancer, that depends from CPU available
* [ ] Implement smart queue algorithm that indexing new sites homepage in higher priority
* [ ] Implement database auto backup on crawl process completing
2 years ago
* [x] Add transactions to prevent data loss on DB crashes
* [x] JSON API
2 years ago
* [ ] Distributed index data sharing between the nodes trough service API
2 years ago
* [x] An idea to make unique gravatars for sites without favicons, because simpler to ident, comparing to ipv6
2 years ago
* [ ] An idea to make some visitors counters, like in good old times?
2 years ago
#### Contributions
2 years ago
2 years ago
Please make a new branch of master|sqliteway tree for each patch in your fork before create PR
2 years ago
```
git checkout master
git checkout -b my-pr-branch-name
```
See also: [SQLite tree](https://github.com/YGGverse/YGGo/tree/sqliteway)
#### Donate to contributors
2 years ago
* @d47081: [BTC](https://www.blockchain.com/explorer/addresses/btc/bc1qngdf2kwty6djjqpk0ynkpq9wmlrmtm7e0c534y) | [DOGE](https://dogechain.info/address/D5Sez493ibLqTpyB3xwQUspZvJ1cxEdRNQ) | Support our server by order [Linux VPS](https://www.yourserver.se/portal/aff.php?aff=610)
2 years ago
#### Feedback
2 years ago
Please, feel free to share your ideas and bug reports [here](https://github.com/YGGverse/YGGo/issues) or use sources for your own implementations.
2 years ago
Have a good time.