Ask questionsFeature request, Duplicate page detector
Do you want to request a feature or report a bug? feature. Adding duplicate link/page detection could be a good idea. Google detects duplicate pages even if the page would not have the canonical references. You are already crawling the page. you can make a hash of the content of each page, store it in an array. whenever you want to add a link to sitemap.xml, first you need to make sure if there is no same hash in the array
Answer questions lgraubner
The crawler should not add duplicate pages.
Related questionsNo questions were found.