Differences Between API and Web Interface results in Google Custom Search Engine

Question

I'm currently working on a project that utilizes the Google Custom Search Engine (CSE) API to retrieve fact-checked information based on user queries. However, I'm encountering a significant issue where the results returned by the API differ substantially from those shown when using the CSE directly via the web interface.

The Problem:

When I run a query through the web interface of my CSE, I get highly relevant results that are exactly what I need for my service. For example, when I search for "Imane boxeuse homme JO femme scandale," the first three URLs returned are directly related to the query and provide the most pertinent information.

However, when I make the same query using the CSE API, the results are different and often include many irrelevant links that do not match the quality or relevance of the results seen in the web interface. This discrepancy is problematic for my service as it lowers the overall quality and reliability of the information provided to my users.

What I’ve tried:

• Matching parameters: I have ensured that the API request includes parameters like hl=fr and gl=fr to match the language and location settings of my CSE, which has helped improve results but has not entirely resolved the issue.
• Sorting by relevance: According to the documentation, using an empty string for sort_by should sort results by relevance, but this hasn't seemed to make a significant difference.
• Parameter verification: I've carefully reviewed the URL and query parameters in both the API and web interface to ensure they are identical.

What I need help with:

I would like to understand why there's a difference in the results between the API and the web interface. Specifically: Why does the API return different (and often less relevant) results compared to the web interface? Are there additional parameters or best practices I should be using to ensure that the API returns the same top results as the web interface? Is there any way to ensure that the API consistently delivers the same high-quality, relevant results as the web interface?

Example to reproduce

Below are some URLs demonstrating the differences in results between the web interface and the API for the same query.

1. Web Interface of the CSE:

Returns 3 highly relevant results Web Interface Search Results : https://cse.google.com/cse?q=Imane%20boxeuse%20homme%20JO%20femme%20scandale&cx=70abf6fcf2e004e88&filter=1

2. Different API calls tried:

Exact same parameters: https://www.googleapis.com/customsearch/v1?q=Imane%20boxeuse%20homme%20JO%20femme%20scandale&key=YOUR_API_KEY_HERE&cx=70abf6fcf2e004e88

Adding filter=1: https://www.googleapis.com/customsearch/v1?q=Imane%20boxeuse%20homme%20JO%20femme%20scandale&key=YOUR_API_KEY_HERE&cx=70abf6fcf2e004e88&filter=1

Adding hl=fr and/or gl=fr: https://www.googleapis.com/customsearch/v1?q=Imane%20boxeuse%20homme%20JO%20femme%20scandale&key=YOUR_API_KEY_HERE&cx=70abf6fcf2e004e88&filter=1&hl=fr&gl=fr

Adding sort_by=”” (as per documentation): https://www.googleapis.com/customsearch/v1?q=Imane%20boxeuse%20homme%20JO%20femme%20scandale&key=YOUR_API_KEY_HERE&cx=70abf6fcf2e004e88&filter=1&hl=fr&gl=fr&sort_by=%22%22

Despite trying these variations, the API does not produce results as relevant as those from the web interface.

Note: I’ve noticed that many of the irrelevant results are included because the snippet contains references to the query, even though the actual link is not directly related to the topic.