How Common Crawl and AWS Open Data built the foundation for the AI revolution

When Gil Elbaz founded Common Crawl in 2007, his goal was to make web-scale crawl data available to researchers and organizations that lacked the resources to operate their own crawling infrastructure. Nearly two decades later, that mission has had impacts far beyond what anyone anticipated.

The Amazon Web Services (AWS) Open Data Sponsorship Program has hosted the Common Crawl open repository of web data at no cost since January 2012. It has become one of the most important sources of training data for the large language models (LLMs) reshaping industries.

A 2024 Mozilla Foundation study found that 64 percent of 47 major LLMs published between 2019–2023 used filtered versions of Common Crawl data for training. The dataset has been cited in over 13,000 research papers.

This is the story of a 14-year collaboration between a small nonprofit and AWS, and how open data infrastructure became the foundation of the AI era.

From 5 billion to 300 billion pages at petabyte scale

Common Crawl began collecting data in 2008 using a custom Hadoop-based crawler with a PageRank implementation. In January 2012, the organization joined the AWS Public Data Sets program—now the AWS Open Data Sponsorship Program—making its corpus of 5 billion web pages available at no cost on Amazon Simple Storage Service (Amazon S3). This meant Common Crawl could focus its limited nonprofit resources on crawling and data quality.

As Common Crawl noted at the time, “Demonstrating their commitment to an open web, AWS hosts public data sets at no charge for the community. Placing our data in the public data sets program not only benefits the larger community, but it also saves us money.”

In 2013, Common Crawl replaced its custom crawler with CCBot, a system based on the Apache Software Foundation’s Nutch web crawler, and adopted the Web ARChive (WARC) format (ISO 28500) as its standard distribution format. In March 2022, Common Crawl celebrated the 10-year anniversary of being a part of AWS Open Data Sponsorship Program and introduced Amazon CloudFront as a new distribution channel alongside Amazon S3, giving users faster access to the data.

Today, the Common Crawl archive contains more than 300 billion web pages, with monthly crawls capturing between 2.3–2.7 billion pages as of this writing.

The LLM inflection point

The release of the OpenAI GPT-3 model in 2020 drew global attention to the role Common Crawl plays in AI development. The GPT-3 paper reported that approximately 82 percent of its raw pre-training tokens were derived from filtered Common Crawl data, the beginning of an industry-wide pattern.

Common Crawl data has been used to train frontier models including GPT-3 (OpenAI, 2020), the BigScience Large Open-science Open-access Multilingual Language Model (BLOOM, 2022), the Meta Llama series, Google Gemini, and Falcon LLM (Technology Innovation Institute, 2023). More recently, Hugging Face FineWeb datasets, NVIDIA Nemotron-CC (6.3 trillion tokens), and Google MADLAD-400 (419 languages) have applied increasingly sophisticated filtering to Common Crawl snapshots.

Scaling infrastructure to meet AI demand

The explosion of AI training workloads placed extraordinary demands on the underlying infrastructure of Common Crawl. By late 2023, downloads had doubled every 6 months for several years, and aggressive downloaders began causing performance disruptions. AWS worked directly with Common Crawl to deploy rate limiting in November 2023, restoring reliable access for the broader community.

Today, the infrastructure spans Amazon S3 as the canonical distribution point, Amazon CloudFront for faster global access, Amazon Athena for querying the index without downloading raw data, and Amazon EMR, AWS Lambda, and Amazon SageMaker for large-scale processing and model fine-tuning. In April 2026, Common Crawl began experimentally distributing data through Hugging Face, although Amazon S3 remains the primary source.

This infrastructure means that a graduate student at a university in Nairobi, a two-person startup in São Paulo, and a research lab in Bangalore can access the same foundational data powering frontier AI systems at no cost.

What this means for AWS customers

The value Common Crawl delivers to the AWS environment extends well beyond the dataset itself.

For builders

If you’re training or fine-tuning language models, you can access pre-training data at scale without building crawling infrastructure, using the derived datasets such as Colossal Clean Crawled Corpus (C4), FineWeb, RefinedWeb, OSCAR, and Nemotron-CC. The Registry of Open Data on AWS lists Common Crawl with tutorials and usage examples that can help you query and process the data without downloading petabyte-scale archives.

For researchers

Over 13,000 published papers have used Common Crawl data across computational linguistics, information retrieval, machine translation, web science, digital preservation, cybersecurity, longitudinal web analytics, phishing detection, health misinformation retrieval, and studies of geolinguistic representation in text data and social science.

You can build on this foundation the same way Stanford’s 2014 Global Vectors for Word Representation (GloVe) word embedding model and Facebook AI Research’s fastText did: both trained word vectors on Common Crawl data covering up to 157 languages.

For organizations

If you’re building search, recommendation, translation, or knowledge extraction systems, you can prototype and validate approaches using Common Crawl before building your own data pipelines.

The Web Graphs give you structural web intelligence, 279.4 million host-level nodes and 13.4 billion edges, plus 122.3 million domain-level nodes and 6.1 billion edges as of the January 2026 release, which are unavailable from other open sources.

For the multilingual AI community

If you’re working to address the overrepresentation of English in AI training data, the Common Crawl Web Languages Project, which was launched December 2024, and the CommonLID (Language Identification) benchmark, released early 2026 and covering 109 languages with MLCommons, EleutherAI, and Johns Hopkins University, provide critical resources.

The CommonLID benchmark was created through community annotation, with native speakers annotating over 350,000 lines of web text. Additionally, the High-Performance Language Technologies (HPLT) project uses Common Crawl data to build a 30-trillion-token multilingual dataset spanning 198 languages, further expanding multilingual AI capabilities beyond English.

Looking ahead

As AI models become more capable and more multilingual, demand for high-quality open training data continues to grow. The AWS Open Data Sponsorship Program helps this foundational resource remain available to everyone, from individual researchers to the world’s largest AI laboratories.

As Rich Skrenta, executive director of the Common Crawl Foundation, puts it: “Common Crawl would not exist without Amazon’s support for open data. We are grateful for our partnership and look forward to serving the AI ecosystem together.” Open data isn’t a side project. It’s infrastructure. And infrastructure, when done right, compounds.

Accessing Common Crawl on AWS

You can access Common Crawl data from the us-east-1 AWS Region at no cost in three formats: WARC (raw crawl data), WAT (metadata and link graphs), and WET (extracted plaintext). Monthly releases include index files in columnar format for querying with Amazon Athena.

For detailed format documentation and access instructions, visit the Common Crawl page on the Registry of Open Data on AWS, or visit the Common Crawl getting started page for different access methods.