Your browser is out of date.
You are currently using Internet Explorer 7/8/9, which is not supported by our site. For the best experience, please use one of the latest browsers.
Traditional proxies simply route packets from Point A to Point B. If you want to transform the data, you usually have to route it to a separate microservice, which introduces serialization bottlenecks and network latency.
const DataStream = require("scramjet"); const axios = require("axios"); // A sample pool of rotating proxy servers const proxyPool = [ host: "192.168.1.50", port: 8080 , host: "192.168.1.51", port: 8080 , host: "192.168.1.52", port: 8080 ]; // Helper function to pick a random proxy function getRandomProxy() const index = Math.floor(Math.random() * proxyPool.length); return proxyPool[index]; // A stream of target URLs to scrape const urlSource = [ "https://example-target.com", "https://example-target.com", "https://example-target.com", ]; // Initialize Scramjet DataStream DataStream.fromArray(urlSource) .map(async (url) => const proxy = getRandomProxy(); try // Stream the HTTP request through the assigned proxy const response = await axios( method: "get", url: url, proxy: host: proxy.host, port: proxy.port , headers: "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" , responseType: "stream" // Keeps the data in a stream state ); return url, stream: response.data ; catch (error) // Handle blocked proxies or dead links gracefully return url, error: error.message ; ) .filter(item => !item.error) // Filter out failed requests .map(async (item) => // Parse the incoming stream data on the fly let rawData = ""; for await (const chunk of item.stream) rawData += chunk; // Simulate extracting specific data point (e.g., Price) const priceMatch = rawData.match(/"price":\s*"([^"]+)"/); return url: item.url, price: priceMatch ? priceMatch[1] : "N/A", timestamp: new Date().toISOString() ; ) .assign( // Pipe the clean object directly to your data destination done: (data) => console.log("Successfully Scraped & Saved:", data) ) .catch(err => console.error("Stream Error:", err)); Use code with caution. Why this design matters:
: Setting it up requires a specific environment, usually involving pnpm installations and the configuration of service workers. Quickstart - Scramjet - Mintlify scramjet proxy
Scramjet Proxy is built differently. It is architected around . Instead of buffering entire payloads, it acts as a true pipe—data flows through it in real-time. This makes it exceptionally powerful for:
# WebSocket support (crucial for streaming data) proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; Traditional proxies simply route packets from Point A
const DataStream, App = require("scramjet");
// Return the app so Scramjet knows to listen for HTTP return app; priceMatch[1] : "N/A", timestamp: new Date()
A proxy acts as an intermediary hub that intercepts, processes, and redirects data flowing from one point to another.
The development of scramjet proxies is an active area of research, with several organizations and companies investing significant resources in this technology. Future directions for scramjet proxy research include:
Scramjet's primary use case is bypassing restrictive network filters in environments such as schools, workplaces, or countries with heavy internet censorship. By rewriting JavaScript on the fly, it circumvents filters that block keywords or use DPI.
The heat transfer system is designed using advanced materials and heat transfer technologies, such as heat pipes and thermal protection systems. The pressure simulation system is designed using advanced pressurization and control systems, such as high-pressure air bottles and control valves.