15 Commits

Author	SHA1	Message	Date
ghost	9916fb701f	implement basic api	2023-04-23 03:01:51 +03:00
ghost	e6b1e8029c	add missed regex replacement rule	2023-04-10 03:18:50 +03:00
ghost	5c8d299a4a	add meta:robots tag support #2	2023-04-09 03:28:31 +03:00
ghost	8e8d89db0e	implement database cleaner	2023-04-09 00:06:28 +03:00
ghost	df6f2a1869	implement CRAWL_ROBOTS_POSTFIX_RULES configuration #5	2023-04-08 22:28:31 +03:00
ghost	2495a2bbc7	implement MySQL/Sphinx data model #3, add basical robots.txt support #2	2023-04-07 04:04:24 +03:00
ghost	c9cd38f6ac	update variable names #2	2023-04-04 01:38:32 +03:00
ghost	ed2d4047b4	implement robots.txt library #2	2023-04-04 00:27:32 +03:00
ghost	e7e4bb686c	fix curl exec double call	2023-04-03 04:47:31 +03:00
ghost	ff95df72c1	implement hostname identicons	2023-04-03 01:30:09 +03:00
ghost	4ea01bf8b4	implement search results pagination	2023-04-02 23:36:35 +03:00
ghost	04dbbc3adf	make url/src column ukeys digital by using crc32	2023-04-02 18:56:56 +03:00
ghost	b218b8bbc3	make url/src columns unique keys, add insert/ignore construction	2023-04-02 18:09:44 +03:00
ghost	d5f33ad643	add ceawl in queue notification	2023-04-02 01:30:50 +03:00
ghost	72985eaf9e	initial commit	2023-04-01 19:29:39 +03:00