Confirmed: ChatGPT uses Google SERP Snippets for its Answers [A Test with Proof]

I’ve run a simple but straight-forward to follow test that confirms the reliance of ChatGPT on Google SERPs snippets for its answers. Let’s go through it: 

1. I Published A New Page in LearningAISearch.com without submitting to any search engine

I published a new indexable LLMS.txt Generators page, linked from the home page, and across the site navigation; however, I didn’t submit it to any search engine after publishing it.

My goal was to assess when and how the page was going to start showing up in ChatGPT answers. So right after publishing it, and before submitting it anywhere, I checked.

2. ChatGPT couldn’t locate the recently published page with web search activated

A bit after publishing the page, I went to ask ChatGPT about the page contents (with the web search feature activated).

We can see how it “searched the web” for the URL of the page, including a search with the “site:” command, but it answered that it “couldn’t locate that exact page on LearningAIsearch” and that the URL I had shared, appeared to “not be publicly indexed” or outdated:

This seemed pretty reasonable because I had just published it and I hadn’t submitted it anywhere yet.

Reasonable right? Not for Gemini! It was able to find it.

3. Gemini didn’t have any issues fetching the content directly from the page to provide an accurate answer, even if this hadn’t yet been indexed by Google

I asked for the same information about the page to Gemini, that didn’t have to wait for the page to be indexed by Google search to give an answer about the page information: it goes and fetches it directly, listing it as a source.

But you might be thinking: Maybe Google had indexed it *very* quickly without you being aware of it?

No. Don’t worry… right after this, I checked that:

4. I confirmed that the page wasn’t yet indexed by Google or Bing so far. So after this, I submitted it through the Google Search Console and Bing Webmaster Tools

After the initial search in both ChatGPT and Gemini from above, I checked both doing a few “site:” + URL searches and checked in the Google Search Console and Bing Webmaster Tools to verify that the page hadn’t been yet indexed when I did the initial search test described in steps 2. and 3.

I could confirm it wasn’t indexed in both, and once I had the confirmation it wasn’t indexed in Google and Bing still at this point, I submitted it for indexing:

5. After a little while, the page was now shown to be indexed in Google via the Search Console but not yet in Bing, however, the page wasn’t yet shown using the “site:” command or adding the URL directly in Google Search Results yet

Google started to show the page as indexed through the Google Search Console very quickly, less than an hour after I submitted it:

While Bing didn’t indexed it after crawling it, an issue that still persists even today (the page is not good enough for Bing, it seems):

However, at this point, even if the page was shown to be indexed in the Google Search Console, it wasn’t yet shown in the Google search results when searching for it with the “site:” command or searching for its own URL. I guess not enough time had yet passed, it was still too early for this:

We can see how Bing also had (and still has) this issue, however, is understandable in this case since they mention how they’ve found problems when crawling the page and is not suitable for indexing via their Webmaster Tools:

6. Now that the page was indexed by Google (but not in Bing), I checked again to see if ChatGPT was able to fetch its content… without luck, still. 

After conforming that Google was now showing the page as indexed in the GSC, I asked again in ChatGPT.

But it answered it wasn’t still finding the page when I asked again about the llms.txt generators listed in it, saying that it had tried to locate the specific page, however, “neither direct access nor search engines yielded results”:

Let’s remember that at this point, the page was:

• In Google: already shown to be indexed via the Google Search Console, however, not shown yet in the SERPs when directly searching for it with a “site:” command
• In Bing: neither shown to be indexed in the Bing Webmaster Tools nor shown in Bing’s SERPs when directly searching for it with a “site:” command

I kept checking for some time and this status didn’t change until…

7. After a while, the page started to be shown in Google Search Results with the “site:” column and when searching for the URL directly 

After a few hours, the page started to get finally shown when searching for it in Google with and without the “site:” command as well as when searching for the URL directly:

All this, while still not being indexed in Bing, this is important to remember. 

So I decide to ask again in ChatGPT to see if anything changed in its answer…

8. When I asked ChatGPT again, it was  now able to provide an “incomplete” answer about the page content based on… (drumroll) its Google Search Result Snippet!

Now, this time, ChatGPT was able to give an answer when I asked about the tools listed in the page, however, it took my attention that the answer was incomplete, giving only a single tool of the four listed in the page; all this while searching for the page via different search variations, including the “site:” command again:

It seemed pretty strange, so I asked in a different way: if it could “fetch” the content of the page.

The answer was eye opening: It said it couldn’t fetch the full, live content of the page, but “based on other sources referencing it”.

So if ChatGPT wasn’t able to have full, live access and was depending on other sources, I wanted to first know the reason, so I asked what was stopping it from accessing it.

The answer again was interesting, because although it didn’t disclosed the reason of the lack of access, it said where it got the information from: “I was able to fetch a snippet from the publicly indexed version of the page, which shows…”:

A snippet from where?

When I followed up asking where was that snippet they grabbed the information being shown, the answer was that it had “located a cached snippet via web search that previews the page content – likely from search engine indexing“.

But I knew the page wasn’t indexed yet in Bing, so it had to be … Google search results? I went to check.

When I compared the text snippet provided by ChatGPT vs the one shown in Google Search Results for the specific Learning AI Search LLMs.txt Generators page, I could confirm it was the same information, as you can see below:

So yes, it’s confirmed: ChatGPT is *somehow* relying on Google Search results snippets for their answers. 

I guess this is yet another reason why traditional SEO is not dead and shows once again how ahead Google is not only with their search index, but the way Gemini is able to just go and fetch the page to give an accurate answer even when the page wasn’t yet indexed by Google still, rather than a incomplete, inaccurate one like ChatGPT, relying on search results snippets from… Google.

9. UPDATE [JULY 25]: The page is now indexed by Bing but ChatGPT says it still doesn’t have access to the page content and its answer content is now based on another variation of a snippet shown in Google SERPs (now from the “site:URL” search) but not Bing

I checked again today to see if anything had changed, and found that the page was now shown to be indexed in Bing via their Webmaster Tools:

However, when searching in Bing directly with a “site:” command along with the page URL, or just the URL, the page wasn’t yet shown or only the home page of the site was, as can be seen below:

Nonetheless, when doing a “site:” search along with the parent directory URL of the page, it was finally shown in the search results. Please note how the description snippet in Bing SERPs of the page doesn’t show the meta description, but a truncated preview of the page content:

So you might be wondering: Now that Bing was finally indexing the page, and also *sometimes* showing it in its SERPs, made any change to ChatGPT answer? Was it the Bing snippet shown now instead of the Google one?

I asked ChatGPT again about the page content, and the answer was interesting because:

• It again provided an answer from a search snippet (not the full content).
• The page snippet provided in the answer was different from the one it had previously shown, saying “here’s what that snippet shows”, featuring different content than it previously did.
• When looking if the new snippet provided in the ChatGPT answer was the one shown in Bing search results in the scenario for which it actually listed the page, I confirmed it wasn’t.. it was taking it from another location.
• When checking again if the snippet was one of those shown in Google search results for one of the searches of the page, I could confirm it was the same one shown for the “site:” along with the URL this time, which is actually showing the meta description text (rather than the one of the URL search only that was used before, which was showing a truncated content preview of the page, not the meta description).

So as you can see again, ChatGPT continues to provide an answer based on a Google Search Results snippet (one from a different query related to the page than before and is now using the meta description unlike before), despite the page being now indexed by Bing, where is showing a different description snippet in those scenarios where is now shown in the SERPs.

Will it stay like that after a few days? I’ll keep you updated.

Need more proof that ChatGPT is using Google Search results? 

This is far from being the only one! The test above complements well what Alexis had identified here a few weeks ago with his own tests with different scenarios.

I’ll keep testing. Stay tuned.

Subscribe to the weekly, free SEOFOMO Newsletter to avoid missing out about my latest AI Search tests as well as staying up to date in AI search and SEO news.