50 Commits

Author	SHA1	Message	Date
yggverse	dd914e0e1b	fix cleanup query	2024-03-20 22:33:11 +02:00
yggverse	36972cab19	implement alter index tool	2024-03-20 21:06:18 +02:00
yggverse	2257ce771f	apply cleaner to the current url configuration	2024-03-20 20:18:55 +02:00
yggverse	3884f375d4	save document body text to index	2024-03-20 19:31:56 +02:00
ghost	1c2e8dafb2	collect keywords from document headers	2024-01-23 02:49:52 +02:00
ghost	cfbc84cbaf	sort queue by rank asc	2024-01-23 02:19:35 +02:00
ghost	db9dc8d4ba	force results to string	2024-01-23 01:55:28 +02:00
ghost	50dc9d315a	add rank field	2024-01-22 22:56:36 +02:00
ghost	6f4abe4729	set crc32url as document id	2024-01-22 22:52:37 +02:00
ghost	93baed4b90	delete deprecated documents with HTTP code not 200 on second scan	2023-12-20 08:44:35 +02:00
ghost	33cc778999	crawl newest pages by rand in queue	2023-12-10 00:29:18 +02:00
ghost	35ad144a9e	add stripos url rules for crawl snaps	2023-12-02 22:15:44 +02:00
ghost	0e06ff3c0f	fix debug message	2023-12-02 21:18:57 +02:00
ghost	51d52dea7d	fix destination name	2023-12-02 20:12:03 +02:00
ghost	87ca594860	add debug levels	2023-12-02 16:04:22 +02:00
ghost	33d657cb72	apply sleep on timeout value provided only	2023-12-02 15:03:51 +02:00
ghost	bc00f0c851	make tmp subfolders storage optimization	2023-12-02 14:39:11 +02:00
ghost	f613b44d3f	disable sort by RAND() in crawler queue	2023-12-02 14:22:50 +02:00
ghost	d3f8d1c0e3	fix result output	2023-11-30 02:59:07 +02:00
ghost	86b20cbc51	add debug output on skip condition	2023-11-30 02:36:25 +02:00
ghost	3306dc1961	add skip url filter by stripos condition	2023-11-30 02:24:02 +02:00
ghost	ee074b684a	add semaphore namespace prefix	2023-11-30 00:51:42 +02:00
ghost	27946ff27c	define missed crc32url field value	2023-11-27 21:03:38 +02:00
ghost	38fbc32151	fix document fields update	2023-11-27 20:55:10 +02:00
ghost	08995e6199	randomize new pages queue	2023-11-27 20:24:46 +02:00
ghost	6a9117757b	reset http code to 404 on page index initiation	2023-11-27 19:44:14 +02:00
ghost	015221eafb	fix semaphore condition #5	2023-11-27 19:34:14 +02:00
ghost	a499c363f6	prevent multi-thread execution #5	2023-11-27 19:31:03 +02:00
ghost	2961045c76	implement index cleaner tool #5	2023-11-27 19:29:17 +02:00
ghost	02dd3649a7	add CURL options that prevent crawl queue stuck	2023-11-27 16:54:26 +02:00
ghost	349f26f5ea	update option name	2023-11-26 21:33:34 +02:00
ghost	133548a98c	fix url check conditions	2023-11-26 20:53:31 +02:00
ghost	6f21cb8bf2	add missed crc32url value	2023-11-25 18:25:43 +02:00
ghost	01437065e3	fix duplicates validation	2023-11-25 18:13:58 +02:00
ghost	dfb2c06738	add crc32url filter	2023-11-25 18:10:23 +02:00
ghost	8a827bfcdf	update settings definition	2023-11-25 17:14:36 +02:00
ghost	a50ef908e2	draft alter index tool	2023-11-25 16:37:58 +02:00
ghost	192e45103d	add index settings support	2023-11-25 16:01:46 +02:00
ghost	4c3038e733	fix processed offset	2023-11-25 13:26:03 +02:00
ghost	10b08215d0	fix data types	2023-11-25 13:25:32 +02:00
ghost	b7444b8f12	add queue offset / limit attributes	2023-11-25 13:19:34 +02:00
ghost	da365c1ab1	fix total condition	2023-11-25 04:59:40 +02:00
ghost	3448eb85f7	implement yggo db migration cli tool	2023-11-25 04:44:07 +02:00
ghost	875382c56e	implement FTP snaps	2023-11-25 03:19:54 +02:00
ghost	72f2fdaeca	change config location	2023-11-25 00:16:08 +02:00
ghost	c6e9ba9d09	implement local storage feature with tar.gz compression	2023-11-24 19:51:43 +02:00
ghost	dc807fe4d5	add url trim	2023-11-24 18:37:25 +02:00
ghost	01753b0557	add crawl queue delay support	2023-11-20 00:06:17 +02:00
ghost	13cf61b42c	fix debug output	2023-11-19 23:34:13 +02:00
ghost	7dfc800a67	initial commit	2023-11-19 23:00:51 +02:00