Vivid Knowledge, the Israeli internet scraping firm that defeated each Meta and Elon Musk’s X in federal courtroom, unveiled a complete AI infrastructure suite Wednesday designed to provide synthetic intelligence techniques unfettered entry to real-time internet information — a functionality the corporate argues Large Tech platforms are attempting to monopolize.
The announcement of Deep Lookup, Browser.ai, and enhanced information assortment protocols represents a dramatic growth for the decade-old firm, which has reworked from a specialised internet scraping service into what CEO Or Lenchner calls “a singular infrastructure layer for AI corporations.” The transfer comes as synthetic intelligence corporations more and more wrestle to entry present internet info wanted to energy chatbots, autonomous brokers, and different AI purposes.
“The intelligence of at this time’s LLMs is not its limiting issue; entry is,” Lenchner stated in an unique interview with VentureBeat. “We’ve spent the final decade preventing for open entry to public internet information, and these new choices carry us to the subsequent chapter in our journey, one characterised by really accessible information and the following rise of contextually-aware brokers.”
The launch follows Vivid Knowledge’s high-profile authorized victories in 2024, when federal judges dismissed lawsuits from each Meta and X alleging the corporate illegally scraped their platforms. These rulings established essential authorized precedent defining what constitutes “public information” on the web — info that may be considered with out logging in and due to this fact might be legally collected and used.
The courtroom instances revealed that each Meta and X had been Vivid Knowledge clients even whereas suing the corporate, highlighting the contradictory stance many tech giants have taken towards internet scraping. The rulings have broader implications for the AI {industry}, which depends closely on internet information to coach and function language fashions.
“It was revealed in courtroom that each of them have been a Vivid Knowledge buyer, as a result of everybody wants information, everybody, particularly those that are constructing fashions,” Lenchner defined. “We’re the one firm that has the monetary assets, and I’d even say the braveness to do this.”
Choose William Alsup, who presided over the X case, wrote that giving social media corporations “free rein to resolve, on any foundation, who can acquire and use information” dangers creating “info monopolies that will disserve the general public curiosity.” The ruling established that information viewable with out login credentials constitutes public info that may be legally scraped.
Vivid Knowledge had beforehand filed a countersuit in opposition to X, alleging the platform violated antitrust legal guidelines by attempting to create a knowledge monopoly to learn Musk’s AI firm, xAI. Nevertheless, that case has since been settled. “Although the phrases confidential, Vivid Knowledge has by no means backed down from its elementary perception that public information ought to be out there to the general public. In step with that perception, we’re happy to report that Vivid Knowledge will proceed to supply the identical industry-leading companies that it at all times has and that our clients have come to count on,” Lenchner stated.
Deep Lookup and Browser.ai goal AI corporations scuffling with information entry
The corporate’s new merchandise tackle what Lenchner identifies because the three core necessities for AI techniques: algorithms, compute energy, and information entry. Whereas Vivid Knowledge doesn’t develop AI algorithms or present computing assets, it goals to turn out to be the definitive resolution for the third requirement.
Deep Lookup features as a pure language analysis engine designed to reply advanced, multi-layered enterprise questions in real-time. Not like general-purpose search engines like google and yahoo or AI chatbots that present summaries, Deep Lookup makes a speciality of complete outcomes for queries starting with “discover all.” For instance, customers can ask for “all transport corporations that went via the Panama and Suez canals in 2023 whose Q3 revenues declined by over 2 p.c.”
The system attracts from Vivid Knowledge’s large internet archive, which at present comprises over 200 billion HTML pages and provides 15 billion month-to-month. By subsequent 12 months, the archive is anticipated to exceed 500 billion pages. “It’s not simply random internet pages, it’s truly what the world cares about, as a result of our 20,000 clients characterize billions of web customers,” Lenchner famous.
Browser.ai represents what the corporate calls “the {industry}’s first unblockable, AI-native browser.” Designed particularly for autonomous AI brokers, the cloud-based service mimics human conduct to entry web sites with out triggering bot detection techniques. It helps pure language instructions and might carry out advanced internet interactions like reserving flights or making restaurant reservations.
The browser infrastructure already processes over 150 million internet actions every day, based on the corporate. “Nearly all of them are clients,” Lenchner stated of AI agent corporations which have raised vital funding. “As a result of what we found out, they usually found out, is that we resolve that downside of getting into an internet site with out being blocked and executing internet actions on the web site.”
MCP Servers (Mannequin Context Protocol) supplies a low-latency management layer enabling AI brokers to go looking, crawl, and extract stay information in real-time. The protocol permits builders to construct AI techniques that may act on present info slightly than relying solely on coaching information.
Patent portfolio and proxy community create aggressive moat in opposition to blocking
Vivid Knowledge’s aggressive benefit stems from what Lenchner describes as an “obsession” with overcoming web site blocking mechanisms. The corporate holds over 5,500 patent claims on its expertise and operates the world’s largest proxy community with greater than 150 million IP addresses throughout 195 nations.
“We have now such an excellent look into the web,” Lenchner defined. “For a very long time now, now we have been mapping the web, and for a very long time now, we’re additionally archiving large chunks of the web.”
The corporate’s strategy entails refined methods to imitate human conduct, utilizing actual units, IP addresses, and browser fingerprints slightly than easy automated scripts. This makes detection and blocking extraordinarily tough for web sites.
“The one method to block us, virtually, is to place the information behind the login, then we received’t even attempt,” Lenchner stated. “Typically there’s a new blocking logic that we received’t resolve instantly. It can take our analysis group 12 hours, three days that’s like essentially the most it was, and we’ll unlock it.”
Income surpasses $100 million as AI demand explodes post-ChatGPT
Whereas Vivid Knowledge stays privately held by a personal fairness agency, Lenchner confirmed with VentureBeat the corporate’s annual recurring income surpassed $100 million a number of years in the past. The enterprise has skilled explosive progress because the launch of ChatGPT in late 2022, as AI corporations scrambled to entry coaching information and real-time info.
“Beginning March 2023, which is just about when GPT-3 modified the world, the AI, or what we name the information for AI, use case simply completely exploded for us as an organization,” Lenchner stated. “All the things else can also be rising, as a result of everybody wants extra information, interval. However this use case is rather like nothing we’ve seen earlier than.”
The corporate serves over 20,000 companies, together with Fortune 500 corporations and main AI laboratories. Conventional clients embrace e-commerce platforms monitoring competitor pricing, monetary companies corporations looking for market intelligence, and enterprises conducting enterprise analysis.
GDPR compliance and moral practices differentiate from rivals
Vivid Knowledge has invested closely in compliance infrastructure to deal with privateness considerations round information assortment. The corporate follows European GDPR and California CCPA laws, mechanically notifying people when their private info is collected from public sources and offering deletion choices.
“The regulation and the laws are clear because the European GDPR and at the very least California and CCPA laws got here to play,” Lenchner defined. “If we collected your e mail tackle, for instance, we’ll mechanically ship you an e mail saying, ‘Hey, that is who we’re. We collected your private info from the general public area. Right here’s an enormous button you may click on if you wish to evaluation it, and you may clearly ask to delete it.’”
The corporate maintains a big compliance group and in depth documentation of its practices, which proved beneficial throughout courtroom proceedings. “We enterprises particularly love us as a result of now we have our moral stand that was scrutinized in US courts twice,” Lenchner stated.
Net entry wars intensify as tech giants search information monopolies
The battle over internet information entry displays broader tensions within the AI {industry} about info management and aggressive benefit. As AI techniques turn out to be extra refined, entry to present, complete internet information turns into more and more beneficial — and contentious.
Lenchner predicts the online will turn out to be “extra closed” over time, just like how Google maintains unique entry to its internet crawling capabilities whereas others should use various companies. “A couple of tech giants are gonna get free entry to each web site with their brokers,” he stated. “The remainder might want to use our infrastructure or another person’s infrastructure.”
The corporate can also be observing new tendencies, together with companies scraping AI chatbots for advertising functions and the emergence of recent protocols like MCP that allow AI brokers to work together with internet companies extra successfully.
“All of those guys which might be consuming large quantities of knowledge, and all of us are utilizing them, it’s all going in the direction of constructing the brains of the robots,” Lenchner stated. “It’s okay that you’ve a chatbot that’s speaking to a human, as a result of that’s ultimately what a robotic will do.”
Robotic brains and agent financial system drive subsequent section of progress
Vivid Knowledge’s transformation from internet scraping service to AI infrastructure supplier displays the quickly evolving wants of the substitute intelligence {industry}. As corporations rush to deploy AI brokers and autonomous techniques, entry to real-time internet information turns into as essential as computing energy and algorithmic sophistication.
The authorized precedents established via Vivid Knowledge’s courtroom victories could show as vital as its technical improvements, probably shaping how your complete AI {industry} accesses and makes use of internet info. With main tech platforms more and more limiting information entry whereas concurrently creating their very own AI techniques, impartial infrastructure suppliers like Vivid Knowledge could turn out to be important for sustaining aggressive stability within the AI ecosystem.
“We’re an infrastructure firm,” Lenchner emphasised. “We’re very gifted engineers that hardly go wherever, simply sit with our computer systems and write code. We’re doing it nicely. We have now no intentions to do anything.”
The Deep Lookup beta launches Tuesday for enterprise clients, with basic public entry out there via a waitlist. Browser.ai and MCP Servers are already out there to enterprise purchasers via Vivid Knowledge’s present platform.
