537 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
ghost	dc1b3a169c	add peak memory usage debug	2023-08-15 09:35:31 +03:00
ghost	e7201c33de	add memory usage debug	2023-08-15 09:21:43 +03:00
ghost	c9a354e4ba	implement hostSetting set/get methods	2023-08-14 12:22:54 +03:00
ghost	b2d7fb2fef	fix line break return	2023-08-14 12:08:05 +03:00
ghost	6085677e67	upgrade yggstate db query	2023-08-11 00:35:03 +03:00
ghost	ab0391e29e	fix url parser path	2023-08-07 14:14:12 +03:00
ghost	f8845c620f	update installation/setup guide	2023-08-07 14:04:32 +03:00
ghost	183ae91ccc	add composer support, refactor FS tree to psr-4	2023-08-07 14:00:13 +03:00
ghost	7bb1eb5b61	add class deprecation notice	2023-08-07 13:22:24 +03:00
ghost	034a683df7	add YGGstate DB crawl integration	2023-08-07 00:13:04 +03:00
ghost	3d9db381e8	fix CRAWL_MANIFEST_API_VERSION	2023-08-06 21:27:56 +03:00
ghost	3c3443b3fd	freeze crawl on remote storage connection lost, infinitely repeat new attempt after 60 seconds until storage connected again	2023-08-06 17:57:42 +03:00
ghost	872ea25d00	remove deprecated condition	2023-08-05 22:00:26 +03:00
ghost	fff75d4d86	update debug message	2023-08-05 21:58:18 +03:00
ghost	6eefd9b762	fix undefined variable	2023-08-05 21:57:11 +03:00
ghost	e953c01eaa	update debug message	2023-08-05 21:55:37 +03:00
ghost	bd212edb97	update debug message	2023-08-05 21:52:26 +03:00
ghost	1b287c8d28	update debug message	2023-08-05 21:40:59 +03:00
ghost	562b97ba8f	update debug message	2023-08-05 21:39:44 +03:00
ghost	c5ae6974bd	fix PDO calls	2023-08-05 21:36:28 +03:00
ghost	b3ec1d42a7	fix empty URI processing	2023-08-05 21:31:33 +03:00
ghost	7ddb47619a	update debug message	2023-08-05 21:17:05 +03:00
ghost	9fe33a3b2c	update CLI roadmap	2023-08-05 21:16:09 +03:00
ghost	6e069a86e5	update readme	2023-08-05 21:11:40 +03:00
ghost	513addc7af	add query totals counting, update crawler debug	2023-08-05 21:03:45 +03:00
ghost	6e03a76ed8	add CURLOPT_SSL_VERIFYHOST/CURLOPT_SSL_VERIFYPEER options	2023-08-05 20:24:47 +03:00
ghost	004a5336de	remove htmls pages ban on title tag not available	2023-08-05 20:01:31 +03:00
ghost	f9774f2431	add innodb_buffer_pool_size default value	2023-08-05 19:51:30 +03:00
ghost	de28d85a71	add connection exceptions	2023-08-05 19:39:49 +03:00
ghost	142d496108	fix SQL syntax error	2023-08-05 19:31:29 +03:00
ghost	d46c4921c5	add page break	2023-08-05 19:24:32 +03:00
ghost	80b33f619c	fix PAGES_LIMIT condition	2023-08-05 19:24:21 +03:00
ghost	d024ffd770	implement unlimited settings customization for each host	2023-08-05 19:06:39 +03:00
ghost	ab6c0379c8	implement hosts crawl queue, move robots, sitemaps, manifests to this task	2023-08-04 09:32:12 +03:00
ghost	6ee5e53ef4	show sitemaps processed debug	2023-08-04 09:07:46 +03:00
ghost	71724ae33f	refactor manifest crawling	2023-08-04 09:00:03 +03:00
ghost	cb37c57bc4	rename example files	2023-08-03 18:49:29 +03:00
ghost	68d5820f30	reserve one hour for huge load operations	2023-08-03 18:47:39 +03:00
ghost	efbbf19601	fix multimedia snaps	2023-08-03 17:41:55 +03:00
ghost	6862fb35cd	update readme	2023-08-03 15:33:34 +03:00
ghost	282a6d609d	update manifest API	2023-08-03 15:31:57 +03:00
ghost	b24d31f360	refactor cleaner, delegate tasks to crawler, init hostSetting table	2023-08-03 15:25:38 +03:00
ghost	fd90e2d517	keep banned pages data	2023-08-03 14:31:06 +03:00
ghost	ab8b6f6315	rename variables	2023-08-03 14:24:37 +03:00
ghost	02612d098b	delete getFoundHostPage method, update API version	2023-08-03 14:08:45 +03:00
ghost	11e02da66d	memory usage optimization, rename methods, remove memchached dependency from the model	2023-08-03 10:48:27 +03:00
ghost	cbabea595b	rename method name	2023-08-03 10:26:37 +03:00
ghost	7e3248ca2c	rename method name	2023-08-03 10:26:14 +03:00
ghost	772975059c	add mysql conf example	2023-08-03 09:25:43 +03:00
ghost	7c407e0d1f	update crontab example	2023-08-03 08:34:51 +03:00