ALL >> General >> View Article
Publishers Unblock Openai’s Crawler As Tiktok’s Parent Bytedance Boasts A 25x Faster Web Scraper
OpenAI has recently secured a massive $6.6 billion in funding, bringing its valuation to $157 billion, a testament to the company's market dominance in generative artificial intelligence (GenAI). Backed by notable investors like Microsoft, NVIDIA, and Thrive Capital, OpenAI has positioned itself as a leader in the AI space. However, even with this influx of capital, OpenAI predicts losses exceeding $5 billion this year. Despite this, the company remains confident, forecasting over $11 billion in revenue for the upcoming year, largely driven by its AI products like ChatGPT.
A crucial component of OpenAI’s growth strategy is its ability to gather data from across the web using its web crawler, GPTBot. This crawler scrapes content to help train its AI models, which require vast datasets to produce high-quality results. Initially, many websites blocked GPTBot from accessing their content, with over 33% of websites implementing restrictions. Yet, recent reports indicate a shift, with the number of sites blocking OpenAI’s crawler dropping to 25%. Additionally, major news publishers have softened their stance, reducing their ...
... block rate from 90% to 50%.
This change in attitude among publishers can partly be attributed to new partnerships OpenAI has formed with key content providers, including TIME, NewsCorp, Reddit, and Condé Nast. These collaborations may offer mutual benefits, allowing publishers to play a role in shaping the future of AI while OpenAI gains access to valuable content. Still, not all publishers who have unblocked GPTBot are doing so willingly. The Onion, for example, unintentionally unblocked the crawler during a recent migration to a new hosting service. The outlet’s CEO, Ben Collins, was quick to dismiss the possibility of partnering with OpenAI, referring to it as a “Plagiarism Machine.”
The broader debate surrounding the use of AI web crawlers is complex. AI models require massive amounts of data for training, but this comes with legal and ethical questions. The use of web scrapers like GPTBot and ByteDance’s Bytespider, which was recently revealed to be even faster than GPTBot, has raised concerns about data privacy and copyright infringement. Moreover, ByteDance’s Bytespider reportedly ignores robots.txt files, which are used by websites to block unauthorized access, causing further alarm.
The issue becomes even more complicated in light of geopolitical tensions between the U.S. and China. ByteDance, the parent company of TikTok, is already under scrutiny for its handling of U.S. data, and the aggressive use of its web crawler has sparked additional worries. As U.S. companies like OpenAI and Chinese firms like ByteDance continue to expand their AI capabilities, questions about who controls access to online data and how that data is used are becoming increasingly urgent.
Publishers now face a critical decision: Should they embrace AI companies and unblock their web crawlers, potentially benefiting from new opportunities? Or should they hold out for better legal protections and assurances before giving AI firms access to their content? The answer to this dilemma may shape the future of both the publishing industry and AI development.
Read More: https://www.techdogs.com/tech-news/td-newsdesk/publishers-unblock-openais-crawler-as-tiktoks-parent-bytedance-boasts-a-25x-faster-web-scraper
Add Comment
General Articles
1. International School In Malaysia | Top International School In MalaysiaAuthor: elisha
2. Sonofit™ | Uk Official | #1 New Hearing Formula
Author: rajesh
3. Essential Benefits Of Digital Marketing Efforts
Author: Anthea Johnson
4. Dissatisfaction Of Even Old-time Customers With Amazon India
Author: Yash Kumar
5. Vidmate App Download For Android
Author: vidmate app
6. Finding The Best Seo Company In Mumbai
Author: Siddhi Dheniya
7. Melasma Laser Treatment For Smooth And Even Skin Tone
Author: pavitra
8. Winter Beauty Package At La Femme Hair, Skin & Bridal Salon At La Femme In Satellite, Ahmedabad
Author: lafemmeindia
9. How To Get Your Clients To Make Faster Payments?
Author: Invoice Temple
10. Discover Why We Are The Best Multispeciality Hospital In Jaipur
Author: YATIKA
11. Gutters, Roofing, Windows, And Sidings | What Comes First?
Author: Gutter Empire LLC
12. The Role Of Technology In Outstation Taxi Bookings
Author: RIDEXPRESS
13. Crystalline Designers
Author: Crystalline
14. Wpc Outdoor Wall Panels
Author: karthik
15. The Importance Of Pharma Hcp Engagement In Modern Healthcare
Author: Jesvira