524 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
ghost	d2469e9adc	fix meta variables overwrite	2023-06-14 02:53:14 +03:00
ghost	0949d7f871	set default encoding	2023-06-14 02:20:09 +03:00
ghost	1d5d5ead5d	fix DomDocument initiation without encoding provided	2023-06-14 02:20:00 +03:00
ghost	fcda6b9885	remove MIME filters from explorer search form	2023-06-13 23:25:17 +03:00
ghost	f3475035c2	show page size in explorer view, hide not available data	2023-06-13 23:20:22 +03:00
ghost	8a747de341	fix HTML/multimedia content detection	2023-06-13 23:09:44 +03:00
ghost	93c6067fd9	fix host page mime detection	2023-06-13 22:29:28 +03:00
ghost	c07d6af52f	add new mime preset	2023-06-13 21:57:01 +03:00
ghost	052b08ea26	show results quantity in the mime filter titles	2023-06-13 21:20:02 +03:00
ghost	80d3912bc7	allow x-raw-image links	2023-06-13 20:26:17 +03:00
ghost	b23f550a1b	skip magnet links	2023-06-13 20:25:37 +03:00
ghost	acba2816e2	remove transaction from tables optimization case	2023-06-13 17:45:02 +03:00
ghost	be81299c84	update readme	2023-06-13 17:35:47 +03:00
ghost	b2cf9fc6a5	do table optimization in separated transaction	2023-06-13 16:51:16 +03:00
ghost	ab78e17ca8	add hostPage.size collection	2023-06-13 12:45:12 +03:00
ghost	830e96b03d	increase minimum requirements	2023-06-13 03:16:29 +03:00
ghost	7892784f5c	add httpCode column to hostPageSnapDownload table	2023-06-12 13:34:25 +03:00
ghost	20726fca45	update readme	2023-06-10 00:20:39 +03:00
ghost	dd736c7923	crontab schedule optimization	2023-06-10 00:19:27 +03:00
ghost	a79993a94b	add an mk	2023-06-08 00:11:02 +03:00
ghost	edec590e09	fix MAYBE filter in the default search mode	2023-06-06 00:36:13 +03:00
ghost	e1fb7f8c17	change query separators to the MAYBE operator in default search mode	2023-06-05 23:33:07 +03:00
ghost	9379809261	colorize meow	2023-06-05 23:08:29 +03:00
ghost	0af5d165d3	remove logCrawler column not in use	2023-06-05 22:06:55 +03:00
ghost	4b16b41440	make transaction for each item in crawl queue	2023-06-05 22:01:22 +03:00
ghost	b585b16d31	fix datatype error detection	2023-06-05 21:02:18 +03:00
ghost	8726512cf0	change morphology from stem_enru to lemmatize_ru_all/lemmatize_en_all	2023-06-05 18:20:49 +03:00
ghost	8bc8a943e7	add lemmatize_de_all	2023-06-05 18:13:31 +03:00
ghost	3d3fcdda87	update readme	2023-06-05 13:44:23 +03:00
ghost	a1249859a8	update readme	2023-06-05 13:42:19 +03:00
ghost	eef23cc830	update readme	2023-06-05 13:40:35 +03:00
ghost	17f69b9661	add min_word_len, min_prefix_len, html_strip, index_exact_words presets example	2023-06-05 13:36:15 +03:00
ghost	1e2736d67b	skip empty mime type index	2023-06-04 18:10:59 +03:00
ghost	c5e25d17fb	prevent page ban when it MIME in the whitelist, skip steps below only (make multimedia/streaming resources visible in search results)	2023-06-04 17:44:09 +03:00
ghost	4fa33afe40	prevent infinitive connection on streaming resources detected	2023-06-04 17:02:32 +03:00
ghost	345c59b5f4	collect target location links on page redirect available	2023-06-04 14:58:33 +03:00
ghost	5d7f2bf68c	fix snap foreign keys deletion	2023-06-04 13:39:47 +03:00
ghost	242e0abd86	ban pages only on data type error codes only	2023-06-04 13:10:32 +03:00
ghost	62a4f33b53	load missed dependency	2023-06-04 12:27:20 +03:00
ghost	512bd56056	ban page that throws the error and stuck the crawl queue	2023-06-04 12:04:41 +03:00
ghost	5a47c66e55	fix readme description	2023-06-04 11:49:25 +03:00
ghost	f49076bb0c	index homepages and shorter URL with higher priority	2023-06-04 11:38:56 +03:00
ghost	3b1590cf7b	update readme	2023-06-01 00:07:11 +03:00
ghost	69eb8d4ed4	update readme	2023-05-31 16:30:05 +03:00
ghost	52bd900ad4	update readme	2023-05-31 16:28:58 +03:00
ghost	982be2a949	add the description text source	2023-05-30 21:46:52 +03:00
ghost	cb60d52a0b	update documentation	2023-05-29 22:36:13 +03:00
ghost	45c4f7b7b0	add database optimization settings	2023-05-29 22:13:41 +03:00
ghost	fc687f7f2c	update readme	2023-05-18 14:13:00 +03:00
ghost	3113057a36	update readme	2023-05-18 14:11:42 +03:00