> For the complete documentation index, see [llms.txt](https://dinproxy.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dinproxy.gitbook.io/docs/guides/web-unlocker/proxy-mode.md).

# Proxy mode

Use the Web Unlocker as an ordinary proxy from any scraper or HTTP client.

Point your scraper or HTTP client at the Web Unlocker as if it were a normal HTTP proxy. Each page you request with `GET` is fetched by the Web Unlocker (rendering, retries, anti-bot checks) and comes back as the response. Nothing else in your code changes.

Billing is the same as the [API](https://dinproxy.gitbook.io/docs/guides/web-unlocker/api-reference.md): **1 result per successful page**, failed pages are free.

On the dashboard, **Products → Web Unlocker → Connect your tools → Proxy** shows everything below already filled in for the proxy user you choose.

## Address and sign-in

| Setting   | Value                                                                                                                                                                       |
| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Proxy     | `unblocker.dinproxy.com:7780` (an HTTP proxy; HTTPS sites go through it as usual)                                                                                           |
| User name | Your [proxy user](https://dinproxy.gitbook.io/docs/getting-started/proxy-users.md), e.g. `1024~myuser`. Add `@country=us` to fetch from a country: `1024~myuser@country=us` |
| Password  | The proxy user's password                                                                                                                                                   |

`@country=` is the only option to add to the user name here.

## HTTPS and the certificate

To fetch an HTTPS page for you, the proxy has to read your request, so it answers the secure connection with its own certificate, issued by the **DINPROXY Web Unlocker** certificate authority. Choose one:

* **Skip the certificate check for this proxy.** `curl -k`, `verify=False` in Python requests, `rejectUnauthorized: false` in Node.js. Scrapy skips the check by default. The [examples](#examples) show it for each language.
* **Trust the DINPROXY certificate in your scraper.** Download it from `https://app.dinproxy.com/api/unblocker/proxy-ca.crt` (or **Download certificate** on the dashboard), then use `curl --cacert dinproxy-unlocker-ca.crt`, `verify="dinproxy-unlocker-ca.crt"` in requests, or `ca` in Node.js.

{% hint style="warning" %}
Trust the certificate only in your scraper. Don't install it in your operating system or browser.
{% endhint %}

The proxy reads what you send (address, cookies, headers) only to pass it to the Web Unlocker, the same as an API request. A page is kept only when your client disconnects before it arrives: the same request within 10 minutes collects it free. With your own `Idempotency-Key`, the page is kept for 24 hours.

## Options

Options are request headers starting with `X-DinProxy-`. They are removed before the request goes out. Their meanings are those of the [API parameters](https://dinproxy.gitbook.io/docs/guides/web-unlocker/api-reference.md).

| Header                        | Values                                                                                 | Default |
| ----------------------------- | -------------------------------------------------------------------------------------- | ------- |
| `X-DinProxy-Render`           | `html` or `png` (a screenshot)                                                         | `html`  |
| `X-DinProxy-JS-Render`        | `true` or `false`                                                                      | `true`  |
| `X-DinProxy-Country`          | Two-letter country code, e.g. `de`                                                     | any     |
| `X-DinProxy-Wait-Ms`          | Extra wait after loading, 0 to 100000. Wait-Ms × (Auto-Runs + 1) may be at most 100000 | —       |
| `X-DinProxy-Wait-Selector`    | Wait for an element matching this CSS selector                                         | —       |
| `X-DinProxy-Block-Resources`  | Any of `image`, `javascript`, `video`, comma-separated without spaces                  | —       |
| `X-DinProxy-Clear`            | Remove `js` and/or `css` from the HTML, comma-separated without spaces                 | —       |
| `X-DinProxy-Follow-Redirects` | `true` or `false`                                                                      | —       |
| `X-DinProxy-Auto-Runs`        | Retries of a failed attempt, 0 to 10                                                   | `2`     |
| `X-DinProxy-Forward-Headers`  | `true` sends your other headers too (see below)                                        | `false` |

* Your **`Cookie`** header is always passed on.
* Your **other headers** (`User-Agent`, `Accept-Language`, your own) are passed on only with `X-DinProxy-Forward-Headers: true`. Without it the Web Unlocker sends browser-like headers itself, which gets through more often than library defaults such as `python-requests/2.32`.
* Use only the `X-DinProxy-` headers listed above, spelled as shown.

## What comes back

A successful page returns **HTTP 200** (or the site's own 2xx status):

* The page as HTML. A page that is JSON or plain text keeps its type (`application/json`, `text/plain`). With `X-DinProxy-Render: png` you get the screenshot as `image/png`.
* Compressed with gzip when your client accepts it.
* `X-DinProxy-Request-Id`: the request's key; `X-DinProxy-Replayed: true` when it was a free repeat; `X-DinProxy-Final-Url`: where the page ended up after redirects.

The site's own response headers (such as `Set-Cookie`) and binary files (images, PDFs) do not come back: the Web Unlocker returns the page's content.

## What to use it with

* **`GET` requests**, one per page.
* **An HTTP client or scraper**, such as the [examples](#examples) below or [Scrapy](#scrapy). For JavaScript-heavy pages keep `X-DinProxy-JS-Render: true` (the default): the Web Unlocker runs the page's JavaScript for you. A browser such as Puppeteer, Playwright or Selenium would fetch dozens of images, scripts and styles for each page, and each would cost a result, so the proxy leaves those out (`403 RESOURCE_NOT_SUPPORTED`, not charged).

## Speed, limits and retries

* A page can take up to **2 minutes**. Set your client's timeout to at least **150 seconds**.
* Up to **100 pages at a time** per account, shared with the API, MCP and the command-line tool. Beyond that you get `429 TOO_MANY_REQUESTS`: try again after a moment.
* **Retrying never pays twice.** If your client times out or loses the connection and repeats the same request (same URL, options and proxy user) within 10 minutes, it receives the page fetched meanwhile without another charge. For longer, send your own `Idempotency-Key` header: it works as in the [API](https://dinproxy.gitbook.io/docs/guides/web-unlocker/api-reference.md) for 24 hours.
* `429`, `502`, `503` and `504` are safe to retry. Scrapy retries them by default. If you send your own `Idempotency-Key`, use a new one after a `502` or `504`.

## Examples

{% tabs %}
{% tab title="Node.js" %}

```javascript
// Save as fetch.mjs, then: npm install undici && node fetch.mjs
import { fetch, ProxyAgent } from "undici";

const agent = new ProxyAgent({
  uri: "http://unblocker.dinproxy.com:7780",
  token: "Basic " + Buffer.from("1024~myuser:YOUR_PASSWORD").toString("base64"),
  requestTls: { rejectUnauthorized: false }, // or { ca: readFileSync("dinproxy-unlocker-ca.crt") }
});

const res = await fetch("https://example.com", { dispatcher: agent, signal: AbortSignal.timeout(150_000) });
console.log(res.status, (await res.text()).length);
```

{% endtab %}

{% tab title="Python" %}

```python
# pip install requests
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import quote
import requests, urllib3
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

USER, PASSWORD = "1024~myuser", "YOUR_PASSWORD"
PROXY = f"http://{quote(USER, safe='')}:{quote(PASSWORD, safe='')}@unblocker.dinproxy.com:7780"
VERIFY = False  # or "dinproxy-unlocker-ca.crt"
urllib3.disable_warnings()

session = requests.Session()
session.proxies = {"http": PROXY, "https": PROXY}
session.mount("https://", HTTPAdapter(pool_maxsize=10, max_retries=Retry(
    total=3, backoff_factor=2, status_forcelist=[429, 502, 503, 504], allowed_methods=["GET"])))

def fetch(url):
    r = session.get(url, verify=VERIFY, timeout=150)
    r.raise_for_status()
    return r.text

urls = ["https://example.com/page-1", "https://example.com/page-2"]
with ThreadPoolExecutor(max_workers=10) as pool:
    for url, html in zip(urls, pool.map(fetch, urls)):
        print(url, len(html))
```

{% endtab %}

{% tab title="cURL" %}

```bash
# One page (-k skips the certificate check)
curl -k -x 'http://unblocker.dinproxy.com:7780' -U '1024~myuser:YOUR_PASSWORD' 'https://example.com'

# With certificate checks on
curl -o dinproxy-unlocker-ca.crt 'https://app.dinproxy.com/api/unblocker/proxy-ca.crt'
curl --cacert dinproxy-unlocker-ca.crt -x 'http://unblocker.dinproxy.com:7780' -U '1024~myuser:YOUR_PASSWORD' 'https://example.com'

# A screenshot from Germany
curl -k -x 'http://unblocker.dinproxy.com:7780' -U '1024~myuser@country=de:YOUR_PASSWORD' \
  -H 'X-DinProxy-Render: png' 'https://example.com' -o page.png
```

In Windows PowerShell, type `curl.exe` instead of `curl`: plain `curl` means something else there.
{% endtab %}

{% tab title="Java" %}

```java
// Java 11+. Save as Main.java and run: java Main.java
import java.net.Authenticator;
import java.net.InetSocketAddress;
import java.net.PasswordAuthentication;
import java.net.ProxySelector;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.security.cert.X509Certificate;
import java.time.Duration;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

public class Main {
    static final String USER = "1024~myuser"; // add @country=us for a country
    static final String PASSWORD = "YOUR_PASSWORD";

    public static void main(String[] args) throws Exception {
        // Java refuses proxy passwords for HTTPS sites unless this is cleared first.
        System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");

        // Skips the certificate check. To keep it on, import dinproxy-unlocker-ca.crt
        // into Java's trust store with keytool and remove .sslContext(ssl) below.
        TrustManager[] trustAll = { new X509TrustManager() {
            public void checkClientTrusted(X509Certificate[] chain, String type) {}
            public void checkServerTrusted(X509Certificate[] chain, String type) {}
            public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; }
        } };
        SSLContext ssl = SSLContext.getInstance("TLS");
        ssl.init(null, trustAll, null);

        HttpClient client = HttpClient.newBuilder()
            .proxy(ProxySelector.of(new InetSocketAddress("unblocker.dinproxy.com", 7780)))
            .authenticator(new Authenticator() {
                @Override
                protected PasswordAuthentication getPasswordAuthentication() {
                    return new PasswordAuthentication(USER, PASSWORD.toCharArray());
                }
            })
            .sslContext(ssl)
            .build();

        HttpRequest request = HttpRequest.newBuilder(URI.create("https://example.com"))
            .timeout(Duration.ofSeconds(150)) // a page can take up to 2 minutes
            .build();
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
        String html = response.body();
        System.out.println(response.statusCode() + " " + response.headers().firstValue("x-dinproxy-request-id").orElse(""));
        System.out.println(html.substring(0, Math.min(200, html.length())));
    }
}
```

{% endtab %}

{% tab title="C#" %}

```csharp
// .NET 6+
using System;
using System.Linq;
using System.Net;
using System.Net.Http;

var handler = new HttpClientHandler
{
    Proxy = new WebProxy("http://unblocker.dinproxy.com:7780")
    {
        // Add @country=us to the user name for a country.
        Credentials = new NetworkCredential("1024~myuser", "YOUR_PASSWORD"),
    },
    // Skips the certificate check. To keep it on, trust dinproxy-unlocker-ca.crt
    // on this computer and remove this line.
    ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator,
};
// A page can take up to 2 minutes.
using var client = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(150) };

var response = await client.GetAsync("https://example.com");
var html = await response.Content.ReadAsStringAsync();
response.Headers.TryGetValues("x-dinproxy-request-id", out var requestId);
Console.WriteLine($"{(int)response.StatusCode} {requestId?.FirstOrDefault()}");
Console.WriteLine(html[..Math.Min(200, html.Length)]);
```

{% endtab %}

{% tab title="PHP" %}

```php
<?php
// Requires the PHP cURL extension.
$curl = curl_init('https://example.com');
curl_setopt_array($curl, [
    CURLOPT_PROXY => 'http://unblocker.dinproxy.com:7780',
    // Add @country=us to the user name for a country.
    CURLOPT_PROXYUSERPWD => '1024~myuser:YOUR_PASSWORD',
    // Skips the certificate check. To keep it on, replace these two lines
    // with CURLOPT_CAINFO => 'dinproxy-unlocker-ca.crt'.
    CURLOPT_SSL_VERIFYPEER => false,
    CURLOPT_SSL_VERIFYHOST => 0,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT => 150, // a page can take up to 2 minutes
]);

$html = curl_exec($curl);
if ($html === false) {
    throw new RuntimeException(curl_error($curl));
}
echo curl_getinfo($curl, CURLINFO_RESPONSE_CODE), PHP_EOL;
echo substr($html, 0, 200), PHP_EOL;
```

{% endtab %}

{% tab title="Go" %}

```go
// Go 1.21+
package main

import (
	"crypto/tls"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func main() {
	proxy := &url.URL{
		Scheme: "http",
		// Add @country=us to the user name for a country.
		User: url.UserPassword("1024~myuser", "YOUR_PASSWORD"),
		Host: "unblocker.dinproxy.com:7780",
	}
	client := &http.Client{
		Timeout: 150 * time.Second, // a page can take up to 2 minutes
		Transport: &http.Transport{
			Proxy: http.ProxyURL(proxy),
			// Skips the certificate check. To keep it on, put
			// dinproxy-unlocker-ca.crt in RootCAs instead.
			TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
		},
	}

	response, err := client.Get("https://example.com")
	if err != nil {
		panic(err)
	}
	defer response.Body.Close()
	body, err := io.ReadAll(response.Body)
	if err != nil {
		panic(err)
	}
	fmt.Println(response.StatusCode, response.Header.Get("X-Dinproxy-Request-Id"))
	fmt.Println(string(body[:min(200, len(body))]))
}
```

{% endtab %}

{% tab title="Ruby" %}

```ruby
require 'net/http'
require 'openssl'

uri = URI('https://example.com')
# Add @country=us to the user name for a country.
http = Net::HTTP.new(uri.host, uri.port, 'unblocker.dinproxy.com', 7780, '1024~myuser', 'YOUR_PASSWORD')
http.use_ssl = true
# Skips the certificate check. To keep it on, use
# http.ca_file = 'dinproxy-unlocker-ca.crt' instead.
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
http.read_timeout = 150 # a page can take up to 2 minutes

response = http.request(Net::HTTP::Get.new(uri))
puts "#{response.code} #{response['x-dinproxy-request-id']}"
puts response.body[0, 200]
```

{% endtab %}
{% endtabs %}

### Scrapy

Scrapy does not check certificates by default, so only the proxy goes in each request:

```python
# scrapy runspider pages.py -o pages.jsonl
from urllib.parse import quote
import scrapy

USER, PASSWORD = "1024~myuser", "YOUR_PASSWORD"
PROXY = f"http://{quote(USER, safe='')}:{quote(PASSWORD, safe='')}@unblocker.dinproxy.com:7780"

class PagesSpider(scrapy.Spider):
    name = "pages"
    start_urls = ["https://example.com/page-1", "https://example.com/page-2"]
    custom_settings = {
        "CONCURRENT_REQUESTS": 50,
        "DOWNLOAD_TIMEOUT": 150,
        "RETRY_HTTP_CODES": [429, 502, 503, 504],
    }

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": PROXY})

    def parse(self, response):
        yield {"url": response.url, "title": response.css("title::text").get()}
```

## Errors

The status says what happened and the `X-DinProxy-Error` header carries the code. Refused and failed requests are not charged. After `504 TIMEOUT`, repeat the same request: if the page arrived meanwhile, you get it without paying again. Errors before the connection to the site is set up (wrong password, no results left, a refused address) show in most clients as *"CONNECT tunnel failed"* or *"Tunnel connection failed"* with the status.

| Status     | Code                                                                                                                                                           | Meaning                                                                                                                |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| 400        | `UNSUPPORTED_OPTION`                                                                                                                                           | The user name has an option other than `@country=`                                                                     |
| 400        | `UNSUPPORTED_PARAMETER`, `INVALID_FORMAT`, `INVALID_COUNTRY`, `INVALID_WAIT`, `INVALID_AUTO_RUNS`, `INVALID_SELECTOR`, `INVALID_PARAMETER`, `COUNTRY_CONFLICT` | An `X-DinProxy-` header is unknown or has a wrong value                                                                |
| 400        | `REQUEST_BODY_NOT_SUPPORTED`                                                                                                                                   | A `GET` with a body                                                                                                    |
| 400        | `INVALID_REQUEST`                                                                                                                                              | The request could not be read                                                                                          |
| 400        | `REQUEST_TOO_LARGE`                                                                                                                                            | Cookies and forwarded headers together are larger than about 60 KB                                                     |
| 400        | `INVALID_HEADER`                                                                                                                                               | One of the headers forwarded with `X-DinProxy-Forward-Headers: true` is not allowed                                    |
| 400        | `INVALID_IDEMPOTENCY_KEY`                                                                                                                                      | Your `Idempotency-Key` is longer than 128 characters or has characters other than letters, digits and `. _ : -`        |
| 402        | `RESULT_BALANCE_EXHAUSTED`                                                                                                                                     | No results left: buy more on the Web Unlocker page                                                                     |
| 403        | `TARGET_NOT_ALLOWED`                                                                                                                                           | Only public sites on ports 80 and 443 can be fetched                                                                   |
| 403        | `RESOURCE_NOT_SUPPORTED`                                                                                                                                       | A browser's image, script, style or font request                                                                       |
| 403        | `ADDRESS_BLOCKED`                                                                                                                                              | Your IP tried 20 different wrong passwords within a day and is blocked for 24 hours. Contact support to lift it sooner |
| 405        | `METHOD_NOT_ALLOWED`                                                                                                                                           | Only `GET` is supported                                                                                                |
| 407        | `AUTH_FAILED`                                                                                                                                                  | Wrong or missing proxy user name or password                                                                           |
| 409        | `IDEMPOTENCY_CONFLICT`, `REQUEST_IN_PROGRESS`                                                                                                                  | See [Retrying safely](https://dinproxy.gitbook.io/docs/guides/web-unlocker/api-reference.md#retrying-safely)           |
| 410        | `RESULT_EXPIRED`                                                                                                                                               | The stored page for this `Idempotency-Key` is older than 24 hours                                                      |
| 429        | `TOO_MANY_REQUESTS`, `TOO_MANY_CONNECTIONS`                                                                                                                    | Too many pages or connections at once: try again shortly                                                               |
| 429        | `TOO_MANY_FAILED_LOGINS`                                                                                                                                       | Too many wrong passwords: wait (see `Retry-After`)                                                                     |
| 502        | `UNBLOCKER_UPSTREAM_FAILED`                                                                                                                                    | The page could not be fetched this time; not charged. Retry (with your own `Idempotency-Key`, use a new one)           |
| 502        | `ACCOUNT_CHANGED`                                                                                                                                              | The proxy user was paused or its password changed during the request; not charged                                      |
| 502 or 504 | `REQUEST_EXPIRED`                                                                                                                                              | The request took too long and was abandoned; not charged. Retry (with your own `Idempotency-Key`, use a new one)       |
| 503        | `UNBLOCKER_UNAVAILABLE`, `SERVICE_UNAVAILABLE`, `GATEWAY_BUSY`                                                                                                 | Temporarily unavailable; retry shortly                                                                                 |
| 504        | `TIMEOUT`                                                                                                                                                      | No answer in time; repeat the same request to collect the page if it arrives                                           |
