mirror of
https://github.com/YGGverse/YGGo.git
synced 2025-01-09 06:17:52 +00:00
280 lines
6.7 KiB
Markdown
280 lines
6.7 KiB
Markdown
### YGGo! - Distributed & Open Source Web Search Engine
|
||
|
||
_Проект присвячується захисникам міста Бахмут_
|
||
|
||
Written by inspiration to explore [Yggdrasil](https://yggdrasil-network.github.io) ecosystem, because of last [YaCy](https://yacy.net/) node there was discontinued.
|
||
This engine also could be useful for crawling regular websites, small business resources, local networks.
|
||
|
||
The project goal - simple interface, clear architecture and lightweight server requirement.
|
||
|
||
#### Overview
|
||
|
||
|
||
|
||
https://github.com/YGGverse/YGGo/tree/main/media
|
||
|
||
#### Online instances
|
||
|
||
* [http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo](http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo)
|
||
* [http://94.140.114.241/yggo/](http://94.140.114.241/yggo)
|
||
|
||
#### Requirements
|
||
|
||
```
|
||
php8^
|
||
php-dom
|
||
php-xml
|
||
php-pdo
|
||
php-curl
|
||
php-gd
|
||
php-mbstring
|
||
php-zip
|
||
php-mysql
|
||
php-memcached
|
||
memcached
|
||
sphinxsearch
|
||
```
|
||
|
||
#### Installation
|
||
|
||
* The web root dir is `/public`
|
||
* Deploy the database using [MySQL Workbench](https://www.mysql.com/products/workbench) project presented in the `/database` folder
|
||
* Install [Sphinx Search Server](https://sphinxsearch.com)
|
||
* Configuration examples presented at `/config` folder
|
||
* Make sure `/storage/cache`, `/storage/tmp`, `/storage/snap` folders are writable
|
||
* Set up the `/crontab` by following [example](https://github.com/YGGverse/YGGo/blob/main/config/crontab.txt)
|
||
|
||
#### JSON API
|
||
|
||
Build third party applications / index distribution.
|
||
|
||
Could be enabled or disabled by `API_ENABLED` option
|
||
|
||
###### Address
|
||
|
||
```
|
||
/api.php
|
||
```
|
||
|
||
##### Search
|
||
|
||
Returns search results.
|
||
|
||
Could be enabled or disabled by `API_SEARCH_ENABLED` option
|
||
|
||
###### Request attributes
|
||
|
||
```
|
||
GET action=search - required
|
||
GET query={string} - optional, search request, empty if not provided
|
||
GET type={string} - optional, filter mime type of available or empty
|
||
GET page={int} - optional, search results page, 1 if not provided
|
||
GET mode=SphinxQL - optional, enable extended SphinxQL syntax
|
||
```
|
||
|
||
##### Hosts distribution
|
||
|
||
Returns hosts collected with fields provided in `API_HOSTS_FIELDS` option.
|
||
|
||
Could be enabled or disabled by `API_HOSTS_ENABLED` option
|
||
|
||
###### Request attributes
|
||
|
||
```
|
||
GET action=hosts - required
|
||
```
|
||
|
||
##### Application manifest
|
||
|
||
Returns node information for other nodes that have same `CRAWL_MANIFEST_API_VERSION` and `CRAWL_URL_REGEXP` conditions.
|
||
|
||
Could be enabled or disabled by `API_MANIFEST_ENABLED` option
|
||
|
||
###### Request attributes
|
||
|
||
```
|
||
GET action=manifest - required
|
||
```
|
||
|
||
#### Search textual filtering
|
||
|
||
##### Default constructions
|
||
|
||
```
|
||
word prefix:
|
||
|
||
yg*
|
||
|
||
operator OR:
|
||
|
||
hello | world
|
||
|
||
operator MAYBE:
|
||
|
||
hello MAYBE world
|
||
|
||
operator NOT:
|
||
|
||
hello -world
|
||
|
||
strict order operator (aka operator "before"):
|
||
|
||
aaa << bbb << ccc
|
||
|
||
exact form modifier:
|
||
|
||
raining =cats and =dogs
|
||
|
||
field-start and field-end modifier:
|
||
|
||
^hello world$
|
||
|
||
keyword IDF boost modifier:
|
||
|
||
boosted^1.234 boostedfieldend$^1.234
|
||
|
||
```
|
||
|
||
##### Extended syntax
|
||
|
||
https://sphinxsearch.com/docs/current.html#extended-syntax
|
||
|
||
Could be enabled with following attributes
|
||
|
||
```
|
||
GET m=SphinxQL
|
||
```
|
||
|
||
#### Roadmap
|
||
|
||
##### Basic features
|
||
|
||
* [x] Web pages full text ranking search
|
||
+ [x] Sphinx
|
||
* [x] Unlimited content MIME crawling
|
||
* [x] Flexible settings compatible with IPv4/IPv6 networks
|
||
* [x] Extended search syntax support
|
||
* [x] Compressed page history snaps with multi-provider storage sync
|
||
+ [x] Local (unlimited locations)
|
||
+ [x] Remote FTP (unlimited mirrors)
|
||
+ [x] Privacy-oriented downloads counting, traffic controls
|
||
|
||
##### UI
|
||
|
||
* [x] CSS only, JS-less interface
|
||
* [x] Unique host ident icons
|
||
* [x] Content MIME tabs (#1)
|
||
* [x] Page index explorer
|
||
+ [x] Meta
|
||
+ [x] Snaps history
|
||
+ [x] Referrers
|
||
* [x] Top hosts page
|
||
* [ ] Safe media preview
|
||
* [ ] Results with found matches highlight
|
||
* [ ] The time machine feature by content snaps history
|
||
|
||
##### API
|
||
|
||
* [ ] Index API
|
||
+ [x] Manifest
|
||
+ [x] Search
|
||
+ [x] Hosts
|
||
+ [ ] Snaps
|
||
* [ ] Context advertising API
|
||
|
||
##### Crawler
|
||
|
||
* [x] Auto crawl links by regular expression rules
|
||
+ [x] Pages
|
||
+ [x] Manifests
|
||
* [x] Robots.txt / robots meta tags support (#2)
|
||
* [x] Specific rules configuration for every host
|
||
* [x] Auto stop crawling on disk quota reached
|
||
* [x] Transactions support to prevent data loss on queue failures
|
||
* [x] Distributed index crawling between YGGo nodes trough manifest API
|
||
* [x] MIME Content-type settings
|
||
* [x] Ban non-condition links to prevent extra requests
|
||
* [x] Debug log
|
||
* [x] Index homepages and shorter URI with higher priority
|
||
* [x] Collect target location links on page redirect available
|
||
* [ ] Host page DOM elements collecting by CSS selectors
|
||
* [ ] Custom settings for each host
|
||
* [ ] XML Feeds support
|
||
+ [x] Sitemap
|
||
+ [ ] RSS
|
||
+ [ ] Atom
|
||
* [ ] Palette image index / filter
|
||
* [ ] Crawl queue balancer, that depends of CPU available
|
||
|
||
##### Cleaner
|
||
|
||
* [x] Deprecated DB items auto deletion / host settings update
|
||
+ [x] Pages
|
||
+ [x] Snaps
|
||
+ [ ] Snap downloads
|
||
+ [x] Manifests
|
||
+ [x] Logs
|
||
+ [x] Crawler
|
||
+ [x] Cleaner
|
||
* [x] Banned resources reset by timeout
|
||
* [x] DB tables optimization
|
||
* [x] Debug log
|
||
|
||
##### CLI
|
||
|
||
* [x] help
|
||
* [x] crontab
|
||
* [x] crawl
|
||
* [x] clean
|
||
* [x] hostPageSnap
|
||
+ [x] repair (not tested)
|
||
+ [x] _sync DB-FS relations_
|
||
+ [x] _FTP_
|
||
+ [x] _localhost_
|
||
+ [x] _delete FS missed in the DB_
|
||
+ [x] _FTP_
|
||
+ [ ] _localhost_
|
||
+ [ ] truncate
|
||
* [x] hostPageDom
|
||
+ [x] generate
|
||
+ [x] truncate
|
||
* [ ] hostPage
|
||
+ [ ] add
|
||
|
||
##### Other
|
||
|
||
* [ ] Administrative panel for useful index moderation
|
||
* [ ] Deployment tools
|
||
* [ ] Testing
|
||
* [ ] Documentation
|
||
|
||
#### Contributions
|
||
|
||
Please make a new branch of main|sqliteway tree for each patch in your fork before create PR
|
||
|
||
```
|
||
git checkout main
|
||
git checkout -b my-pr-branch-name
|
||
```
|
||
|
||
See also: [SQLite tree](https://github.com/YGGverse/YGGo/tree/sqliteway)
|
||
|
||
#### Donate to contributors
|
||
|
||
* @d47081: [BTC](https://www.blockchain.com/explorer/addresses/btc/bc1qngdf2kwty6djjqpk0ynkpq9wmlrmtm7e0c534y) | [DOGE](https://dogechain.info/address/D5Sez493ibLqTpyB3xwQUspZvJ1cxEdRNQ) | Support our server by order [Linux VPS](https://www.yourserver.se/portal/aff.php?aff=610)
|
||
|
||
#### License
|
||
* Engine sources [MIT License](https://github.com/YGGverse/YGGo/blob/main/LICENSE)
|
||
* HTML parser [simple_html_dom](http://sourceforge.net/projects/simplehtmldom/)
|
||
* Home page animation by [alvarotrigo](https://codepen.io/alvarotrigo/pen/GRvYNax)
|
||
* CLI logo by [patorjk.com](https://patorjk.com/software/taag/#p=display&f=Slant&t=YGGo!)
|
||
|
||
#### See also
|
||
|
||
* [YGGwave ~ The Radio Catalog](https://github.com/YGGverse/YGGwave)
|
||
|
||
#### Feedback
|
||
|
||
Please, feel free to share your ideas and bug reports [here](https://github.com/YGGverse/YGGo/issues) or use sources for your own implementations.
|
||
|
||
Have a good time! |