1
0
mirror of https://github.com/YGGverse/YGGo.git synced 2025-01-12 15:58:00 +00:00
YGGo! Distributed Web Search Engine
Go to file
2023-04-02 23:49:04 +03:00
config implement search results pagination 2023-04-02 23:36:35 +03:00
crontab make url/src column ukeys digital by using crc32 2023-04-02 18:56:56 +03:00
library implement search results pagination 2023-04-02 23:36:35 +03:00
media add media files 2023-04-01 21:40:37 +03:00
public fix empty request title 2023-04-02 23:49:04 +03:00
.gitignore add .gitignore 2023-04-02 01:31:20 +03:00
LICENSE Initial commit 2023-04-01 15:15:22 +03:00
README.md Update README.md 2023-04-02 23:37:09 +03:00

YGGo! - Open Source Web Search Engine

Written by inspiration to discover Yggdrasil ecosystem, because of last YaCy node there was discontinued. This engine also could be useful for crawling regular websites, small business resources, local networks.

The project goal - simple interface, clear architecture and lightweight server requirement.

Online examples

http://[201:23b4:991a:634d:8359:4521:5576:15b7]/yggo
http://94.140.114.241/yggo/

Screenshotes

Search page

https://github.com/d47081/YGGo/tree/main/media

License

Engine sources MIT License
Home page animation by alvarotrigo

Requirements

php8^
php-dom
php-pdo
php-curl
sqlite / fts5

Installation

  • The webroot dir is /public
  • Single configuration file placed here /config/app.php.txt and need to be configured and renamed to /config/app.php
  • By the idea, script automaticaly generates database structure in /storage folder (where could be nice to collect other variative and tmp data - like logs, etc). Make sure storage folder writable.
  • Set up the /crontab/crawler.php script for execution every the minute, but it mostly related of the configs and targetal network volume, there is no debug implemented yet, so let's silentize it by /dev/null
  • Script has no MVC model, because of super simple. It's is just 2 files, and everything else stored incapsulated in /library classes.

Roadmap / ideas

  • Web pages full text ranking search
  • Make search results pagination
  • Improve yggdrasil links detection, add .ygg domain zone support
  • Make page description visible - based on the cached content dump, when website description tag not available, add condition highlights
  • Images search (basically implemented but requires testing and some performance optimization)
  • Deprecated pages index cleaner (404 http codes etc)
  • Crawl queue balancer, that depends from CPU available
  • Implement smart queue algorithm that indexing new sites homepage in higher priority
  • Implement database autobackup on crawl process completting
  • Add transactions to prevent data loss on DB crashes
  • Distributed index data sharing between the nodes trough service API
  • An idea to make unique gravatars for sites without favicons, because simpler to ident, comparing to ipv6
  • An idea to make some visitors counters, like in good old times?

Donate to contributors

@d47081 time spent BTC | DOGE

Feedback

Please, feel free to share your ideas and bug reports here or use sources for your own implementations.

Have a good time.