Ask questionsFeature request, Duplicate page detector

Do you want to request a feature or report a bug? feature. Adding duplicate link/page detection could be a good idea. Google detects duplicate pages even if the page would not have the canonical references. You are already crawling the page. you can make a hash of the content of each page, store it in an array. whenever you want to add a link to sitemap.xml, first you need to make sure if there is no same hash in the array


Answer questions lgraubner

The crawler should not add duplicate pages.


Related questions

No questions were found.
Github User Rank List