✓ Verified 💻 Development ✓ Enhanced Data

Markdown Fetch

Optimizes web fetching by using Cloudflare's Markdown for Agents, reducing token consumption by ~80%

Rating
3.9 (109 reviews)
Downloads
22,959 downloads
Version
1.0.0

Overview

Optimizes web fetching by using Cloudflare's Markdown for Agents, reducing token consumption by ~80%.

Complete Documentation

View Source →

Markdown Fetch - 网页抓取优化

背景

Cloudflare 推出 Markdown for Agents 功能:

  • AI 请求时返回 Markdown 格式
  • Token 消耗比 HTML 减少约 80%

使用方法

在需要网页抓取时,使用优化后的 fetch 函数:

javascript
const { optimizedFetch } = require('./markdown-fetch');

const result = await optimizedFetch('https://example.com');
// result.markdown - Markdown 内容(如果有)
// result.html - HTML 内容(备用)
// result.tokensSaved - 节省的 tokens(如果有)

核心逻辑

javascript
async function optimizedFetch(url, options = {}) {
  const headers = {
    'Accept': 'text/markdown, text/html',
    ...options.headers
  };

  const response = await fetch(url, { ...options, headers });
  
  const contentType = response.headers.get('content-type');
  const xMarkdownTokens = response.headers.get('x-markdown-tokens');
  
  let result = {
    url,
    contentType,
    tokensSaved: xMarkdownTokens ? parseInt(xMarkdownTokens) : null
  };
  
  if (contentType.includes('text/markdown')) {
    result.markdown = await response.text();
    result.format = 'markdown';
  } else {
    result.html = await response.text();
    result.format = 'html';
  }
  
  return result;
}

响应处理

Content-Type处理方式
text/markdown直接使用,跳过 HTML 解析
text/html走原有解析逻辑

可选:x-markdown-tokens 日志

如果响应中有 x-markdown-tokens header,记录到日志:

javascript
if (result.tokensSaved) {
  console.log(`[Markdown Fetch] Token 节省: ${result.tokensSaved}`);
}

改动范围

  • 找到所有 HTTP 请求(fetch/axios/request)
  • 统一添加 header
  • 响应处理加判断

测试验证

找一个 Cloudflare 托管的网站测试:

bash
curl -H "Accept: text/markdown, text/html" https://cloudflare-example.com

确认收到 content-type: text/markdown 响应。

Installation

Terminal bash

openclaw install markdown-fetch
    
Copied!

💻Code Examples

}

.txt
## 响应处理

| Content-Type | 处理方式 |
|--------------|----------|
| text/markdown | 直接使用,跳过 HTML 解析 |
| text/html | 走原有解析逻辑 |

## 可选:x-markdown-tokens 日志

如果响应中有 `x-markdown-tokens` header,记录到日志:

}

.txt
## 改动范围

1. 找到所有 HTTP 请求(fetch/axios/request)
2. 统一添加 header
3. 响应处理加判断

## 测试验证

找一个 Cloudflare 托管的网站测试:
example.js
const { optimizedFetch } = require('./markdown-fetch');

const result = await optimizedFetch('https://example.com');
// result.markdown - Markdown 内容(如果有)
// result.html - HTML 内容(备用)
// result.tokensSaved - 节省的 tokens(如果有)
example.js
async function optimizedFetch(url, options = {}) {
  const headers = {
    'Accept': 'text/markdown, text/html',
    ...options.headers
  };

  const response = await fetch(url, { ...options, headers });
  
  const contentType = response.headers.get('content-type');
  const xMarkdownTokens = response.headers.get('x-markdown-tokens');
  
  let result = {
    url,
    contentType,
    tokensSaved: xMarkdownTokens ? parseInt(xMarkdownTokens) : null
  };
  
  if (contentType.includes('text/markdown')) {
    result.markdown = await response.text();
    result.format = 'markdown';
  } else {
    result.html = await response.text();
    result.format = 'html';
  }
  
  return result;
}
example.js
if (result.tokensSaved) {
  console.log(`[Markdown Fetch] Token 节省: ${result.tokensSaved}`);
}

Tags

#web_and-frontend-development #web

Quick Info

Category Development
Model Claude 3.5
Complexity Multi-Agent
Author howtimeschange
Last Updated 3/10/2026
🚀
Optimized for
Claude 3.5
🧠

Ready to Install?

Get started with this skill in seconds

openclaw install markdown-fetch