.Google.com's Gary Illyes discloses Googlebot gathers hyperlinks instead of "complying with" all of them, opposing main paperwork.Googlebot gathers web links as opposed to observing them in real-time.Gary Illyes discussed this info in Google's 'Look Off The Document' podcast.Google.com's records negates the actual spider actions.