6 Commits

Author	SHA1	Message	Date
James	340e78de13	crawler takes request options in constructor, update docs	2020-02-23 18:23:36 +00:00
James	adaaaef3c2	more comprehensive tests and tidy up	2020-02-23 01:19:31 +00:00
James	fe45abaf49	make baseurl (internal links only) optional in crawler constructor	2020-02-22 13:57:43 +00:00
James	c81551e2bf	add tests	2020-02-21 10:59:58 +00:00
James	ac7b849036	make observer collate all found on urls	2020-02-21 10:24:34 +00:00
James	6a986a62c0	inital	2020-02-20 17:06:44 +00:00