YGGo! Distributed Web Search Engine
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

226 lines
5.4 KiB

2 years ago
### YGGo! - Distributed & Open Source Web Search Engine
2 years ago
2 years ago
_Проект присвячується захисникам міста Бахмут_
2 years ago
2 years ago
Written by inspiration to explore [Yggdrasil](https://yggdrasil-network.github.io) ecosystem, because of last [YaCy](https://yacy.net/) node there was discontinued.
2 years ago
This engine also could be useful for crawling regular websites, small business resources, local networks.
2 years ago
2 years ago
The project goal - simple interface, clear architecture and lightweight server requirement.
2 years ago
2 years ago
#### Overview
2 years ago
![Home page](https://github.com/YGGverse/YGGo/blob/main/media/main-sm.png?raw=true)
2 years ago
https://github.com/YGGverse/YGGo/tree/main/media
2 years ago
2 years ago
#### Online instances
* [http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo](http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo)
* [http://94.140.114.241/yggo/](http://94.140.114.241/yggo)
2 years ago
#### Requirements
```
2 years ago
php8^
php-dom
2 years ago
php-pdo
2 years ago
php-curl
2 years ago
php-gd
php-mysql
2 years ago
sphinxsearch
2 years ago
```
#### Installation
2 years ago
2 years ago
* The web root dir is `/public`
2 years ago
* Deploy the database using [MySQL Workbench](https://www.mysql.com/products/workbench) project presented in the `/database` folder
2 years ago
* Install [Sphinx Search Server](https://sphinxsearch.com)
* Configuration examples are placed at `/config` folder
* Make sure `/storage` folder is writable
* Set up the `/crontab` scripts by following [example](https://github.com/YGGverse/YGGo/blob/main/config/crontab.txt)
2 years ago
#### JSON API
2 years ago
Build third party applications / index distribution.
2 years ago
2 years ago
Could be enabled or disabled by `API_ENABLED` option
2 years ago
###### Address
```
/api.php
```
##### Search
Returns search results.
2 years ago
2 years ago
Could be enabled or disabled by `API_SEARCH_ENABLED` option
###### Request attributes
```
2 years ago
GET action=search - required
GET query={string} - optional, search request, empty if not provided
GET type={string} - optional, filter mime type of available or empty
2 years ago
GET page={int} - optional, search results page, 1 if not provided
GET mode=SphinxQL - optional, enable extended SphinxQL syntax
```
##### Hosts distribution
2 years ago
Returns node hosts collected with fields provided in `API_HOSTS_FIELDS` option.
2 years ago
2 years ago
Could be enabled or disabled by `API_HOSTS_ENABLED` option
###### Request attributes
```
2 years ago
GET action=hosts - required
```
##### Application manifest
Returns node information.
Could be enabled or disabled by `API_MANIFEST_ENABLED` option
###### Request attributes
```
GET action=manifest - required
```
2 years ago
#### Search textual filtering
##### Default constructions
2 years ago
```
operator OR:
hello | world
operator MAYBE:
hello MAYBE world
operator NOT:
hello -world
strict order operator (aka operator "before"):
aaa << bbb << ccc
exact form modifier:
raining =cats and =dogs
field-start and field-end modifier:
^hello world$
keyword IDF boost modifier:
boosted^1.234 boostedfieldend$^1.234
```
##### Extended syntax
https://sphinxsearch.com/docs/current.html#extended-syntax
Could be enabled with following attributes
```
GET m=SphinxQL
```
2 years ago
#### Roadmap
2 years ago
2 years ago
##### Basic features
* [x] Web pages full text ranking search
2 years ago
* [x] Unlimited content type groups
* [x] Safe proxy images preview
2 years ago
* [x] Extended syntax support
* [x] Flexible settings compatible with IPv4/IPv6 networks
##### UI
2 years ago
* [x] CSS only, JS-less interface
2 years ago
* [x] Unique ident icons for sites without favicons
* [ ] Results with found matches highlight
* [ ] Content genre tabs (#1)
* [ ] The time machine feature by content history cache preview
* [ ] Link clicks counter
##### API
2 years ago
* [ ] Index API
2 years ago
+ [x] Manifest
+ [x] Search
+ [x] Hosts
+ [ ] MIME list
2 years ago
* [ ] Context advertising API
##### Crawler
* [x] Auto crawl links by regular expression rules
+ [x] Pages
+ [x] Manifests
2 years ago
* [x] Robots.txt / robots meta tags support (#2)
* [x] Specific rules configuration for every host
2 years ago
* [x] Auto stop crawling on disk quota reached
2 years ago
* [x] Transactions support to prevent data loss on queue failures
2 years ago
* [x] Distributed index crawling between YGGo nodes trough manifest API
2 years ago
* [x] MIME Content-type settings
* [x] Ban non-condition links to prevent extra requests
* [x] Debug log
2 years ago
* [x] History snaps
2 years ago
* [ ] Indexing new sites homepage in higher priority
2 years ago
* [ ] Redirect codes extended processing
* [ ] Palette image index / filter
* [ ] Crawl queue balancer, that depends of CPU available
##### Cleaner
* [x] Deprecated DB items auto deletion / host settings update
+ [x] Pages
+ [x] Manifests
+ [x] Logs
+ [x] Crawler
+ [x] Cleaner
2 years ago
* [x] Deprecated history snaps removing
* [x] Banned resources reset by timeout
* [x] Debug log
2 years ago
##### Other
2 years ago
* [ ] Administrative panel for useful index moderation
2 years ago
* [ ] Deployment tools
2 years ago
#### Contributions
2 years ago
2 years ago
Please make a new branch of master|sqliteway tree for each patch in your fork before create PR
2 years ago
```
git checkout master
git checkout -b my-pr-branch-name
```
See also: [SQLite tree](https://github.com/YGGverse/YGGo/tree/sqliteway)
#### Donate to contributors
2 years ago
* @d47081: [BTC](https://www.blockchain.com/explorer/addresses/btc/bc1qngdf2kwty6djjqpk0ynkpq9wmlrmtm7e0c534y) | [DOGE](https://dogechain.info/address/D5Sez493ibLqTpyB3xwQUspZvJ1cxEdRNQ) | Support our server by order [Linux VPS](https://www.yourserver.se/portal/aff.php?aff=610)
2 years ago
2 years ago
#### License
* Engine sources [MIT License](https://github.com/YGGverse/YGGo/blob/main/LICENSE)
* Home page animation by [alvarotrigo](https://codepen.io/alvarotrigo/pen/GRvYNax)
#### Feedback
2 years ago
Please, feel free to share your ideas and bug reports [here](https://github.com/YGGverse/YGGo/issues) or use sources for your own implementations.
2 years ago
Have a good time.