IndexError: list index out of range

[danielg-everc]

Hi Im trying to use search_by_url method and Im getting the following:
IndexError: list index out of range

full error
File , line 4
1 from googlelens import GoogleLens
2 lens = GoogleLens()
----> 4 search_result = lens.search_by_url("https://ae01.alicdn.com/kf/S52a81a68ba6c4dd2ba057a7e019092d6f.jpg" target="_blank" rel="noopener noreferrer">https://ae01.alicdn.com/kf/S52a81a68ba6c4dd2ba057a7e019092d6f.jpg")
5 print(search_result)

 in GoogleLens.__get_prerender_script(self, page)
 32 soup = BeautifulSoup(page, 'html.parser')
 34 # Find the script containing 'AF_initDataCallback' with specific key and hash values

---> 35 prerender_script = list(filter(
36 lambda s: (
37 'AF_initDataCallback(' in s.text and
38 re.search(r"key: 'ds:(\d+)'", s.text).group(1) == "0"),
39 soup.find_all('script')
40 ))[0].text
42 # Clean up the script content to prepare it for JSON parsing
43 prerender_script = prerender_script.replace(
44 "AF_initDataCallback(", "").replace(");", "")

[DLFP-socsci]

Hi there,

I've been encountering similar issues when using the search_by_url method. After some debugging, particularly by logging the available <script> elements in the HTML response, it appears that Google has changed the structure of its HTML, making it incompatible with the current parsing logic.

Observations:

• The soup.find_all('script') no longer seems to contain any script matching the criteria AF_initDataCallback.
• This leads to an IndexError because the list(filter(...))[0] logic fails when the filter returns an empty list.

Debugging Steps Taken:

1. Logged all <script> elements from the response to identify any changes in the structure or content.
2. Confirmed in almost all cases that the expected AF_initDataCallback script is missing in the HTML returned by Google.

Proposed Solution:

The logic for extracting data from the AF_initDataCallback script needs to be updated to handle the new HTML structure. Specifically, we need to:

1. Verify whether AF_initDataCallback is still used by Google.
2. If it's been replaced, identify the new mechanism for data embedding.
3. Modify the script-filtering logic in the __get_prerender_script method accordingly.

In the meantime, adding a more robust fallback mechanism or a check for empty results before accessing [0] can prevent the IndexError and provide better error handling. For example:

prerender_scripts = list(filter(
    lambda s: 'AF_initDataCallback(' in s.text and re.search(r"key: 'ds:(\d+)'", s.text),
    soup.find_all('script')
))

if not prerender_scripts:
    raise ValueError("Could not find the expected AF_initDataCallback script. The HTML structure may have changed.")

prerender_script = prerender_scripts[0].text

[DCLASSIFIED]

Google stopped using AF_initDataCallback here's a updated method to extract the json. I haven't figured out how to properly parse it though.

def __get_prerender_script(self, page: str):
    """
    Extracts items from the Google Lens data with proper script cleaning.
    """
    soup = BeautifulSoup(page, 'html.parser')
    scripts = soup.find_all('script')
    data_script = next((s for s in scripts if '(function(){var m=' in s.text), None)
    
    if not data_script:
        return []
    
    js_text = data_script.text
    start_index = js_text.find('(function(){var m=')
    if start_index == -1:
        return []
        
    js_text = js_text[start_index:]
    js_text = js_text.replace('(function(){var m=', '')
    
    # Remove trailing window scripts
    end_index = js_text.find(';window.')
    if end_index != -1:
        js_text = js_text[:end_index]
    js_text = js_text.rstrip('});')
    js_text = js_text.split('};')[0] + '}'
    
    try:
        data = json.loads(js_text)
        return data
        
    except json.JSONDecodeError as e:
        print("Error:", e)
        print("Offending section:", js_text[e.pos-100:e.pos+100])
        return []

[Dentordev]

@DCLASSIFIED Seems they got smarter I had a parser that could do that part and worked quite well but unfortunately google is getting smarter with attempting to stop me. This script I had worked before up until now. My theory is that it's using a redirect of some kind.

        data = ApiResponseParser.__get_prerender_script(src_response)
        # print(data)
        image_sources = []

        for k, v in data.items():
            try:
                if not isinstance(v, str):
                    image_url = v[1][2][0]
                    hr_url = v[1][3][0]

                    if isinstance(v[1][6], dict): 
                        item = v[1][6].popitem()[1]
                        page_url = item[2]
                        img_title=item[3]
                        image_sources.append(SearchItem(page_url, img_title, image_url, hr_url))
            except (IndexError, KeyError, TypeError):
                continue
        return image_sources

I plan to look into reverse engineering the garbled mess it generates and trying something similar to what youtube-dl does with it's JSInterp Module to bypass these problems. Help with Figuring out those portions is appreciated as I am working on a sever backend simillar to searx and whoogle that does it.

[Dentordev]

I have figured out that it's using very sophisticated payloads to try to obfuscate where it is exactly Finding a way to simulate the decryption is what I plan to do but in the meantime I will make a small repo for some of these codes that I've found.