Eighty-six percent of data collection professionals believe AI adoption in web scraping saves time and resources as the demand for web intelligence continues to rise. This is one of the key findings from the new Oxylabs whitepaper, Meeting the Increasing Business Demand for Web Intelligence, which demonstrates the growth and impact of web intelligence solutions.
In cooperation with Censuswide, Oxylabs surveyed 506 software engineers and web scraping professionals from UK and US companies, gathering insights on the issues faced when collecting web intelligence and the impact it brings to businesses. Seventy-seven percent of respondents stated that they had tried using AI-powered web scraping solutions, with those in the US more likely to use AI for web scraping than those in the UK (56% vs. 38%).
The discrepancy might partly be explained by another finding of the survey. The share of professionals who fix parsers every day as a result of changing website structures was found to be greater in the US at 38% over 24% in the UK. The more scraping professionals face such an intense parsing schedule, the more likely they are to turn to AI for assistance.
The web intelligence community has welcomed AI innovations and is prepared for the change these innovations can bring. Only 4% of the respondents stated that AI did not meet their expectations. Meanwhile, 86% of those who tried AI solutions for web scraping confirmed that it helped them save time and resources. Almost half of scraping professionals (47%) use AI-powered solutions regularly, with 77% having tried using AI solutions and another 20% claiming that they are keen to try them, leaving only 3% of web data collection professionals who have no interest in utilising AI.
Juras Juršėnas, Chief Operations Officer at Oxylabs, said: “In many industries, the true extent of AI’s potential impact and its ability to provide a return on the huge investment it had already accumulated is still up for debate. However, large-scale web data collection seems to be perfectly suited for AI enhancements. Building and maintaining data pipelines consists of multiple operations and repetitive tasks that AI solutions can automate or improve. While we are still at an early stage of AI adoption, it is both exciting and vital for web scraping companies to closely watch the emerging AI/ML innovations and trends of leveraging these tools.”
When asked about the web scraping tasks respondents used AI to complete, 67% identified building parsers, while 59% used AI for target unblocking and 52% for building crawling logic. This suggests that AI adopters in web scraping are testing the hypothesis that AI can streamline many if not most, tech tasks. Forty-three percent of scraping professionals stated AI is best suited for target unblocking, suggesting that AI is perceived as most applicable in dealing with sophisticated tasks.
“Web unblocking is one of the first areas where we have utilised AI technologies,” added Aleksandras Šulženko, Product Owner at Oxylabs. “However, while web unblocking becomes a challenge in specific, although important, cases, parsing is a crucial part of every scraping pipeline. Thus, an AI solution for building parsers is even more widely applicable and brings constant value to the customer.
