CoMP 1.0: A Standard for Getting Paid Before AI Crawls Your Content

For most of the web's history, the deal between publishers and crawlers was simple: let search engines index your pages, and they send you readers. AI systems broke that deal. They crawl content to train models and to answer questions directly, and they often send little traffic back. Publishers have responded by blocking crawlers, signing individual licensing deals, or both. What has been missing is a shared, machine-readable way to say "here are my terms" before the crawling starts.

On March 10, the IAB Tech Lab released version 1.0 of its Content Monetization Protocol, known as CoMP, for public comment. Comments are open until April 9.

What CoMP does

CoMP is designed to give content owners and AI systems a standard way to:

• Communicate what content is on offer before any crawling happens
• Establish commercial agreements before content is used
• Signal permissions and licensing terms consistently
• Work across both direct licensing relationships and third-party marketplaces

In short, it aims to turn "blocked" or "allowed" into a structured conversation about terms. IAB Tech Lab CEO Anthony Katsur framed it this way in the announcement: information "is the only input in that equation that does not yet have a consistent commercial infrastructure around it."

Supporters quoted at launch included The Weather Company, People Inc., Bertelsmann, Beeler.Tech and Mobian.

What CoMP assumes

This is the part most coverage skips. CoMP assumes publishers already have effective access controls in place, typically at the edge through a CDN or similar infrastructure. The protocol creates a path from restriction to a commercial market. It does not do the restricting for you.

That matters because a licensing protocol is only as strong as the ability to enforce it. If AI crawlers can access your content freely, they have little reason to negotiate. If you can identify and control access, the protocol gives you a standard way to offer terms. The full specification and its supporting documents are on the CoMP initiative page, and it is worth having both your engineering and business teams read it.

Why B2B and specialist publishers should care most

General news is widely available from many sources, and AI companies can often substitute one outlet for another. Specialist content is different. Trade publications, B2B research, technical documentation, industry pricing data, regulatory analysis and professional reference material are often unique, expensive to produce, and exactly what AI systems need to answer professional questions accurately.

That gives B2B publishers leverage, if they can use it:

• Your content may be more valuable per page to an AI system than general consumer content.
• Your audience is small, so each lost visit costs more in advertising and lead generation revenue.
• Your advertisers are paying for context, and that context is exactly what AI summaries reproduce without the ad.

How this fits with advertising revenue

Licensing is not a replacement for advertising. For most publishers it will be an additional revenue line, and for many it will be small at first. But it changes how you should think about the content that drives your ad business. Pages that answer specific professional questions are both your best inventory for contextual advertising and your most exposed content to AI answers. Protecting their value is protecting your ad revenue.

What to do now

1. Know who is crawling you

Review server and CDN logs for AI crawler activity. Identify which bots visit, how often, and which sections they hit. Many publishers are surprised by the volume.

2. Get control at the edge

Robots.txt signals intent, but it depends on crawlers choosing to respect it. Look at what your CDN or hosting provider offers for bot identification and access control. CoMP explicitly assumes this layer exists.

3. Classify your content by value

Separate content that is unique and expensive to produce from content that is widely available. Your licensing posture should differ between them. Blanket decisions leave money on the table in one direction or the other.

4. Read the spec and comment

The comment window is open until April 9. If you have views on how terms should be expressed, how marketplaces should work, or what verification publishers need, now is the time to say so. Standards shaped without publisher input tend to serve publishers less well.

5. Keep ad operations in the conversation

Licensing decisions often sit with executives or legal teams. Make sure the people who run your ad business are involved, since changes to crawler access and content exposure affect traffic, and traffic drives ad revenue.

What to watch

• Adoption by AI companies. A protocol for commercial terms only works if the buyers use it. Watch which AI developers commit to supporting it.
• Marketplace support. Third-party marketplaces that aggregate content licensing could make CoMP practical for smaller publishers without legal teams.
• How it connects to existing controls. Publishers will want CoMP to work alongside robots.txt, CDN-level controls and other emerging licensing signals rather than replace them.

The takeaway

CoMP will not solve the AI traffic problem on its own, and it depends on publishers having real control over access. But it is a meaningful step toward a market where content used by AI systems carries a price, set in advance and communicated in a standard way. For B2B and specialist publishers, whose content is both scarce and valuable, that market could matter a great deal. The preparation, understanding who crawls you, controlling access and knowing what your content is worth, is useful regardless of how the standard evolves.