71 Commits

Author	SHA1	Message	Date
ghost	13431008c4	add options documentation	2023-04-23 03:16:54 +03:00
ghost	9916fb701f	implement basic api	2023-04-23 03:01:51 +03:00
ghost	81cb970248	add options documentation	2023-04-23 01:54:10 +03:00
ghost	8da150b295	add options documentation	2023-04-23 01:46:34 +03:00
ghost	8f09db5045	add options documentation	2023-04-23 01:32:34 +03:00
ghost	c4dfb58fe3	add options documentation	2023-04-23 01:14:31 +03:00
ghost	8e8d89db0e	implement database cleaner	2023-04-09 00:06:28 +03:00
ghost	df6f2a1869	implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5	2023-04-08 22:28:31 +03:00
ghost	8d102ecdf7	index hosts with enabled status only	2023-04-08 18:23:48 +03:00
ghost	0b12e872a3	add host name to the search index	2023-04-08 18:22:53 +03:00
ghost	e98146b78b	index only 200 http code pages	2023-04-07 05:34:45 +03:00
ghost	0f2b772fa8	remove not indexed pages from the search index	2023-04-07 04:50:01 +03:00
ghost	2495a2bbc7	implement MySQL/Sphinx data model #3, add basical robots.txt support #2	2023-04-07 04:04:24 +03:00
ghost	a07ca1dce1	add ipv6 example	2023-04-04 01:39:48 +03:00
ghost	79663c84db	add CRAWL_META_ONLY option	2023-04-03 03:07:54 +03:00
ghost	ff95df72c1	implement hostname identicons	2023-04-03 01:30:09 +03:00
ghost	4ea01bf8b4	implement search results pagination	2023-04-02 23:36:35 +03:00
ghost	c770a912f0	fix crawl request warnings	2023-04-02 18:14:42 +03:00
ghost	aadfe7f551	add env-less option	2023-04-02 18:08:03 +03:00
ghost	1f9b1503b9	add server environment configuration to keep the multi-adresing support	2023-04-02 01:51:54 +03:00
ghost	72985eaf9e	initial commit	2023-04-01 19:29:39 +03:00