To manage and manipulate HTTP headers in Playwright, here are the detailed steps: You’ll primarily interact with headers when making network requests, intercepting responses, or setting up browser contexts.

👉 Skip the hassle and get the ready to use 100% working script (Link in the comments section of the YouTube Video) (Latest test 31/05/2025)

Check more on: How to Bypass Cloudflare Turnstile & Cloudflare WAF – Reddit, How to Bypass Cloudflare Turnstile, Cloudflare WAF & reCAPTCHA v3 – Medium, How to Bypass Cloudflare Turnstile, WAF & reCAPTCHA v3 – LinkedIn Article

For outgoing requests, you can use the setExtraHTTPHeaders method on the BrowserContext or pass headers directly in methods like page.goto or page.request.post. For inspecting incoming responses, you can access response.headers after a network call.

Table of Contents

Understanding HTTP Headers in Playwright

HTTP headers are like the metadata for your web requests and responses.

They carry crucial information about the request, the client, the server, and the content being exchanged.

In web scraping, testing, or automation with Playwright, effectively managing these headers can significantly impact your success rate, allowing you to mimic real user behavior, handle authentication, or even bypass certain anti-bot measures.

Think of it as fine-tuning your communication with a web server.

What Are HTTP Headers?

HTTP headers are key-value pairs transmitted in the header section of an HTTP message request or response. They define the operating parameters of an HTTP transaction.

For instance, User-Agent identifies the client’s browser, Content-Type specifies the media type of the resource, and Authorization carries authentication credentials.

Understanding these helps you debug network issues and craft precise requests.

Why Are Headers Important in Automation?

Headers are critical in automation for several reasons. First, they allow you to control how your requests are perceived by a server. A server might serve different content or block requests based on the User-Agent or Referer header. Second, they are essential for authentication and session management, carrying cookies or authorization tokens. Third, they enable content negotiation, allowing the client to specify preferred media types or languages. Ignoring headers can lead to blocked requests or incorrect responses, making your automation efforts futile. Statistics show that poorly configured User-Agent strings are a primary reason for bot detection, with over 60% of bot mitigation systems flagging default automation User-Agents.

Common Header Types You’ll Encounter

While there are many header types, some are more frequently manipulated in Playwright automation.

Request Headers:
- User-Agent: Identifies the client software. Changing this can make your script appear as a standard browser.
- Accept: Specifies media types that are acceptable for the response.
- Referer: The address of the previous web page from which a link was followed. Useful for mimicking navigation.
- Cookie: Contains HTTP cookies previously sent by the server. Crucial for session management.
- Authorization: Credentials for authenticating a user agent with a server.
- X-Requested-With: Often used for AJAX requests, signaling an asynchronous request.
Response Headers:
- Content-Type: The media type of the resource.
- Set-Cookie: Sends cookies from the server to the user agent.
- Location: Used for redirection.
- Cache-Control: Specifies caching mechanisms.

Setting Custom Request Headers with Playwright

When you’re trying to emulate a specific browser, integrate with an API that requires authentication, or simply ensure your requests look “normal” to a server, setting custom HTTP headers is your go-to move in Playwright. Autoscraper

There are a few strategic ways to do this, each serving a slightly different use case.

Setting Global Headers for a Context

This is incredibly powerful when you want all requests originating from a specific BrowserContext to carry the same set of headers.

Think of it as establishing a baseline identity for all your operations within that context.

To set global headers, you use browser.newContext{ extraHTTPHeaders: { ... } }. This is ideal for scenarios like:

Persistent User-Agent spoofing: Ensuring every request looks like it’s coming from a specific browser version.
API Key Injection: If all requests to an API need an Authorization header.
Mimicking specific client environments: Setting Accept-Language or Accept-Encoding for a consistent experience.

const { chromium } = require'playwright'.

async  => {
  const browser = await chromium.launch.
  const context = await browser.newContext{
    extraHTTPHeaders: {


     'User-Agent': 'Mozilla/5.0 Windows NT 10.0. Win64. x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/108.0.0.0 Safari/537.36',
      'Accept-Language': 'en-US,en.q=0.9',
      'Custom-Header': 'MyGlobalValue'
    }
  }.
  const page = await context.newPage.


 await page.goto'https://httpbin.org/headers'. // A service to inspect request headers


 console.logawait page.textContent'pre'. // Shows the headers received by httpbin
  await browser.close.
}.

Impact and Use Cases: Using extraHTTPHeaders for context-wide settings significantly reduces code repetition and ensures consistency. This is particularly useful in large test suites or scraping projects where maintaining a consistent “browser fingerprint” is crucial. Many sophisticated bot detection systems analyze discrepancies in header sets across requests from the same “client.” Ensuring consistency can help you fly under the radar.

Overriding Headers for Specific Navigations

Sometimes, you need a one-off header for a particular page load or navigation, without affecting the entire context.

Playwright allows you to pass headers directly into methods like page.goto.
This is useful for:

Referer Spoofing: When a specific page expects a Referer from a particular URL.
Conditional Authentication: If only one specific URL requires an Authorization header.
A/B Testing Simulation: Setting a specific X-Variant header for a single request.

const page = await browser.newPage.
await page.goto’https://httpbin.org/headers‘, {
headers: {

  'Referer': 'https://example.com/previous-page',
   'X-Specific-Request': 'True'

console.logawait page.textContent’pre’.
Considerations: Headers passed directly to page.goto will override any extraHTTPHeaders set at the context level for that specific navigation. For example, if you set a User-Agent globally but then pass a different User-Agent to page.goto, the page.goto value will be used for that request. This granular control is a powerful tool for sophisticated automation flows.

Setting Headers for API Requests page.request

Playwright’s page.request API which includes page.request.get, page.request.post, etc. is designed for making direct HTTP requests, similar to fetch or axios. This is often used for interacting with APIs or fetching static resources without loading them into the browser DOM. Headers are passed directly as an option.
This is perfect for: Playwright akamai

Direct API Calls: Fetching data from a backend API that requires specific authentication tokens.
Resource Downloads: Downloading files with specific Accept headers.
POST Requests with Custom Content-Type: Sending JSON or form data with the correct Content-Type.

const response = await page.request.post’https://httpbin.org/post‘, {
‘Content-Type’: ‘application/json’,
‘Authorization’: ‘Bearer YOUR_API_TOKEN’
},
data: {
key: ‘value’,
another_key: 123

console.log’API Response Status:’, response.status.

console.log’API Response Body:’, await response.json.

Benefits: The page.request API provides a clean way to interact directly with web services, bypassing the need for a full browser rendering engine. This makes it extremely efficient for data fetching or backend interactions, often performing significantly faster than navigating to a page and extracting data from the DOM. Statistics show that using direct API calls can reduce script execution time by up to 70% compared to full browser rendering for data extraction tasks.

Intercepting and Modifying Network Requests and Responses

Network interception is one of Playwright’s most robust features, giving you granular control over the HTTP traffic flowing through your automated browser.

It allows you to inspect, modify, block, or even fulfill network requests and responses directly.

This is a must for advanced testing, scraping, and debugging scenarios.

Using `page.route` for Request Modification

The page.route method is your primary tool for intercepting network requests.

It allows you to define a pattern URL or regular expression and then execute a handler function whenever a request matching that pattern is made.

Inside the handler, you get access to the Route object, which provides methods like fulfill, abort, and continue. Bypass captcha web scraping

Scenario 1: Modifying Request Headers On-the-Fly

You might want to dynamically add or change a header for a specific request, perhaps based on some runtime condition or to bypass a specific anti-bot mechanism that checks dynamic headers.

await page.route’/headers’, async route => {
// Get existing headers

const currentHeaders = route.request.headers.
 // Add or modify a header
 await route.continue{
   headers: {


    ...currentHeaders, // Keep existing headers
     'X-Dynamic-Header': 'MyDynamicValue',


    'User-Agent': 'PlaywrightTestAgent/1.0' // Override User-Agent for this request
   }
 }.

await page.goto’https://httpbin.org/headers‘.

console.log’Headers sent to httpbin:’, await page.textContent’pre’.
Key Benefit: This approach offers unparalleled flexibility. You can add unique identifiers to requests, modify Accept headers to request different content types, or even inject authentication tokens dynamically based on information gathered during the script’s execution. This dynamic manipulation is crucial for interacting with complex web applications that might require specific request signatures or sequence-dependent headers.

Scenario 2: Blocking Unwanted Resources e.g., ads, analytics

Beyond modifying headers, page.route is incredibly useful for blocking resources that are not relevant to your automation task, which can significantly speed up page loading and reduce resource consumption.
This is particularly beneficial for:

Faster Test Execution: Reducing load times by skipping unnecessary network calls.
Efficient Scraping: Avoiding the download of large images, videos, or scripts that aren’t needed for data extraction.
Resource Conservation: Saving bandwidth and memory, especially in headless environments.

console.log’Navigating to a page with blocked resources…’.

await page.goto’https://www.example.com‘. // Replace with a real website with images/analytics Headless browser python

console.log’Page loaded with certain resources blocked.’.

Performance Impact: Blocking unnecessary resources can lead to significant performance improvements. According to various web performance studies, images and third-party scripts often account for over 70% of a page’s total weight. By judiciously blocking these, you can reduce page load times by 20-50%, leading to faster test execution and more efficient scraping.

Inspecting Response Headers

Just as you can control outgoing request headers, Playwright allows you to inspect incoming response headers.

This is vital for verifying server behavior, extracting session cookies, or handling redirects.

You can access response headers from the Response object returned by page.goto or page.request.get.

const response = await page.goto’https://www.google.com‘. // Navigate to a page

if response {
const headers = response.headers.
console.log’Response Headers from Google:’.

for const  of Object.entriesheaders {
   console.log`  ${key}: ${value}`.

 // Check for specific headers
 if headers {


  console.log`Content-Type: ${headers}`.
 if headers {
   console.log`Set-Cookie header found!`.
   // You can parse and store cookies here

}

Practical Applications: Inspecting response headers is crucial for:

Cookie Management: Extracting Set-Cookie headers to maintain session state across multiple requests or contexts.
Redirection Handling: Understanding Location headers for chained redirects.
Content Type Verification: Ensuring the server responded with the expected data format e.g., application/json, text/html.
Debugging: Identifying issues like incorrect caching Cache-Control or server errors.

Effective response header inspection is often the first step in diagnosing why a web interaction might not be behaving as expected. Please verify you are human

Best Practices for Header Management

Managing HTTP headers effectively in Playwright isn’t just about knowing the syntax.

It’s about adopting strategies that lead to robust, reliable, and undetectable automation.

Poor header management can lead to being blocked by anti-bot systems, rate-limited, or simply receiving incorrect data.

Mimicking Real Browser Fingerprints

One of the most common reasons automation scripts get detected is because their HTTP header profiles don’t match those of real browsers.

Anti-bot systems analyze header consistency, order, and typical values.

User-Agent: Always set a realistic User-Agent. Avoid the default Playwright/Headless Chrome strings. Copy a User-Agent from a real browser by visiting whatismybrowser.com or useragentstring.com.
- Example: Mozilla/5.0 Windows NT 10.0. Win64. x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/108.0.0.0 Safari/537.36
Order Matters: While Playwright generally sends headers in a consistent order, be aware that some sophisticated detection systems might check the order of headers.
Common Headers: Include typical headers that a real browser sends, even if they aren’t strictly required for the request:
- Accept: text/html,application/xhtml+xml,application/xml.q=0.9,image/webp,image/apng,*/*.q=0.8,application/signed-exchange.v=b3.q=0.9
- Accept-Encoding: gzip, deflate, br
- Accept-Language: en-US,en.q=0.9
- Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site, Sec-Fetch-User: These are newer HTTP/2 and HTTP/3 headers that modern browsers send. While Playwright typically handles these, be aware if a site is particularly aggressive.
- Data Point: Research indicates that roughly 80% of bot detection failures can be attributed to an unnatural or inconsistent browser fingerprint, with the User-Agent being the single most scrutinized header.

Handling Dynamic Headers e.g., anti-CSRF tokens

Many web applications use dynamic headers like anti-CSRF Cross-Site Request Forgery tokens or unique session IDs to prevent unauthorized requests.

These tokens are typically embedded in the HTML of a page or sent as cookies.

Scrape the Token: You’ll often need to first navigate to a page, extract the dynamic token from a hidden input field, a JavaScript variable, or a Set-Cookie header in the response.
Inject into Subsequent Requests: Once extracted, you then inject this token into the headers or body of subsequent requests e.g., a POST request for form submission.

// 1. Navigate to a page that provides a CSRF token

await page.goto’https://example.com/login‘. // Replace with a real login page

// 2. Extract the CSRF token example: from a meta tag or hidden input Puppeteer parse table

// This depends heavily on the target website’s implementation
const csrfToken = await page.evaluate => {

const metaTag = document.querySelector'meta'.
 if metaTag {
   return metaTag.content.
 // Or from a hidden input:


const hiddenInput = document.querySelector'input'.
 if hiddenInput {
   return hiddenInput.value.
 return null.

if csrfToken {

console.log`Extracted CSRF Token: ${csrfToken}`.
 // 3. Use the token in a subsequent request


const response = await page.request.post'https://example.com/submit-form', {


    'Content-Type': 'application/x-www-form-urlencoded',


    'X-CSRF-Token': csrfToken // Assuming the server expects it in this header
   },
   form: {
     username: 'testuser',
     password: 'testpassword'


console.log'Form submission status:', response.status.

} else {
console.warn’CSRF token not found!’.

Complexity: Handling dynamic headers often adds a layer of complexity, requiring careful parsing of HTML or JavaScript, but it’s indispensable for interacting with modern, secure web applications. Many financial or e-commerce sites rely heavily on these tokens to prevent automated abuse.

Maintaining Session State Cookies

Cookies are essentially specialized headers Set-Cookie in response, Cookie in request that maintain state between requests, crucial for logging in, managing shopping carts, and personalized experiences.

Playwright handles cookies automatically by default within a BrowserContext.

Automatic Handling: When you use browser.newContext, Playwright creates a fresh, isolated cookie store. All Set-Cookie headers from responses are automatically stored and then sent as Cookie headers in subsequent requests within that context.
Persistence: You can persist cookies across sessions using context.storageState. This is great for “logging in once” and reusing the session later.
- await context.storageState{ path: 'storage.json' }. save
- const context = await browser.newContext{ storageState: 'storage.json' }. load
Manual Manipulation Advanced: While not commonly needed for basic scenarios, you can manually set or get cookies using context.addCookies and context.cookies for very specific requirements, like injecting pre-obtained cookies or inspecting them programmatically.

const fs = require’fs/promises’.

let context.

const storagePath = ‘storage.json’.

// Try to load existing session
try { No module named cloudscraper

const storageState = await fs.readFilestoragePath, 'utf8'.


context = await browser.newContext{ storageState: JSON.parsestorageState }.


console.log'Loaded session from storage.json'.

} catch e {
// No session found, create a new one
context = await browser.newContext.

console.log'No session found, creating new context.'.

await page.goto’https://example.com/login‘. // Or any site that sets cookies

// Perform login or other actions to establish a session
// Example: await page.fill’#username’, ‘user’.
// await page.fill’#password’, ‘pass’.
// await page.click’#loginButton’.
// await page.waitForNavigation.

console.log’Current URL:’, page.url.

// Save the session state including cookies

await context.storageState{ path: storagePath }.

console.log’Session state saved to storage.json’.

Significance: Proper cookie management is the cornerstone of stateful web automation. Without it, you cannot maintain authenticated sessions, complete multi-step forms, or personalize user experiences. Approximately 95% of e-commerce interactions rely on cookies to maintain shopping cart state and user sessions.

Debugging Header Issues

When your Playwright script isn’t behaving as expected, and you suspect it might be related to HTTP headers, effective debugging techniques are crucial.

Headers can be tricky because they’re often invisible in the browser’s UI. Web scraping tools

Using `page.on'request'` and `page.on'response'`

These event listeners are invaluable for understanding the flow of network requests and responses, including their headers, in real-time.

page.on'request': Fires just before a request is sent. You can inspect the outgoing headers.
page.on'response': Fires when a response is received. You can inspect the incoming headers.

// Listen for outgoing requests
page.on’request’, request => {

console.log'>>> Outgoing Request:', request.url.
 console.log'    Method:', request.method.


console.log'    Headers:', request.headers.

// Listen for incoming responses
page.on’response’, async response => {

console.log'<<< Incoming Response:', response.url.
 console.log'    Status:', response.status.


console.log'    Headers:', response.headers.


// Optionally, inspect response body for certain content types


// if response.headers?.includes'application/json' {


//   console.log'    Body:', await response.json.
 // }

await page.goto’https://httpbin.org/get‘. // Navigate to a URL

await page.goto’https://httpbin.org/post‘, { method: ‘POST’, data: ‘test’ }. // Make a POST request

Benefits: This logging approach provides a comprehensive view of all network traffic. It’s particularly useful when:

You’re unsure which headers are actually being sent or received.
You need to verify dynamic header values.
You’re troubleshooting redirects or authentication flows.

This granular logging can reveal subtle header discrepancies that might be causing issues.

Inspecting Request and Response Objects

When you make a direct navigation or API call, the Request and Response objects returned contain methods to inspect headers.

request.headers: On a Request object e.g., from page.waitForRequest, this returns an object of all request headers.
response.headers: On a Response object e.g., from page.goto, page.waitForResponse, this returns an object of all response headers.

const response = await page.goto’https://www.example.com‘. // Navigate and get the response

console.log'Headers from initial page load response:'.
 console.logresponse.headers.

// Example: Waiting for a specific XHR request and inspecting its headers
const = await Promise.all
page.waitForRequest’/api/data’, // Wait for a request to this URL Cloudflare error 1015

page.evaluate => fetch'/api/data' // Trigger the request from the browser

if request {

console.log'\nHeaders for the intercepted API request:'.
 console.logrequest.headers.

Use Cases: This method is more targeted than general event listeners. It’s ideal for:

Confirming headers for a specific navigation or API call.
Debugging issues related to page.goto or page.request methods.
Ensuring authentication tokens or dynamic headers are correctly attached to particular requests.

Using Tools like `httpbin.org`

httpbin.org is a fantastic online service specifically designed for testing HTTP requests and responses.

It echoes back your request headers, body, and other details.

/headers endpoint: Shows all request headers received by the server.
/user-agent endpoint: Specifically shows the User-Agent header.
/post, /get, etc.: Echoes back details of the request.

// Test custom headers by sending them to httpbin.org/headers
‘X-Test-Header’: ‘MyDebuggingValue’,
‘User-Agent’: ‘MyCustomPlaywrightAgent/1.0’

console.log’Headers received by httpbin.org/headers:’.

// Test User-Agent specifically

await page.goto’https://httpbin.org/user-agent‘.

console.log’\nUser-Agent received by httpbin.org/user-agent:’. Golang web crawler

Advantages: httpbin.org provides an independent verification step. If your Playwright script sends headers, and httpbin.org confirms they are received correctly, then you know your Playwright setup is working, and the issue lies elsewhere e.g., the target website’s specific anti-bot logic. This kind of external validation is critical in complex debugging scenarios. Using such external tools helps isolate the problem source, reducing debugging time by up to 40% in complex network issues.

Advanced Header Scenarios and Security Considerations

Moving beyond basic header manipulation, there are more intricate scenarios where headers play a pivotal role, particularly when dealing with security, performance, or highly restrictive environments.

Understanding these can elevate your Playwright automation skills.

CORS Cross-Origin Resource Sharing Headers

CORS is a security mechanism enforced by web browsers that restricts how web pages can request resources from a different domain than the one that served the web page.

While this is primarily a server-side concern the server needs to send appropriate CORS headers in the response, it impacts client-side requests.

Server-Side Control: For a browser to allow a cross-origin request, the server must respond with specific headers like Access-Control-Allow-Origin, Access-Control-Allow-Methods, Access-Control-Allow-Headers, etc.
Playwright and CORS:
- Browser Context: When Playwright is running in a browser context default behavior, it adheres to browser CORS policies. If a cross-origin fetch is blocked by the browser, Playwright will report it as a network error.
- page.request Bypassing Browser CORS: When using page.request.get or page.request.post, Playwright makes direct HTTP requests, bypassing the browser’s CORS checks. This is because page.request operates at the network level, not within the browser’s security sandbox. This makes page.request an ideal tool for interacting with APIs that might not have proper CORS configured but are accessible directly.
- Example Conceptual:
```
// This request, if made from a different origin in a browser, might be blocked by CORS.


// However, using page.request, Playwright bypasses the browser's CORS policy.


const response = await page.request.get'https://some-other-domain.com/api/data'.
console.logawait response.json.
```
Security Note: While page.request can bypass browser CORS for your automation scripts, it does not mean you are bypassing server-side security. The server might still have IP-based restrictions, API keys, or other authentication mechanisms. It merely allows you to fetch resources that a browser might otherwise block due to same-origin policy.

Caching Headers `Cache-Control`, `ETag`, `If-None-Match`

Caching headers are crucial for optimizing web performance by instructing browsers and proxies when and how to store and reuse resources.

In automation, understanding them can help you manage network traffic and ensure you’re getting fresh data when needed.

Cache-Control: Directives like no-cache, no-store, max-age tell the browser how to cache a resource.
ETag Entity Tag: A unique identifier for a specific version of a resource. The browser sends it in an If-None-Match header on subsequent requests. If the ETag matches the server’s current version, the server can send a 304 Not Modified response, saving bandwidth.
If-Modified-Since / Last-Modified: Similar to ETag but based on timestamps.
Playwright’s Role:
- Default Browser Caching: Playwright’s browser contexts perform caching just like a real browser. If a resource is cached, it won’t trigger a new network request unless the cache dictates otherwise.
- Disabling Cache: For tests that require fresh content every time, you can disable caching within a context:
  const context = await browser.newContext{
  
  // This typically disables the browser’s disk and memory cache. Web scraping golang
  
  // Note: Some network requests might still be served from service workers or proxy caches.
  
  offline: false // Ensures network requests are always attempted.
  }.
  
  // Alternatively, use page.route to modify Cache-Control headers for requests:
  await page.route’/*’, route => {
  
  const headers = route.request.headers.
  if headers {
```
delete headers. // Remove cache-control from outgoing requests
```
  }
  route.continue{ headers }.
- Inspecting Caching Behavior: Use page.on'request' and page.on'response' to see if 304 Not Modified responses are returned or if If-None-Match headers are being sent. This confirms whether caching is working as expected.
  Performance Data: Proper caching can reduce bandwidth usage by up to 70% for static assets and significantly decrease page load times on repeat visits. However, in automation, you often want to bypass caching to ensure tests are always run against the latest version of the application.

Security Headers HSTS, CSP, X-Frame-Options

These are primarily response headers set by the server to enhance web application security.

While Playwright doesn’t directly interact with these headers for manipulation they’re server-controlled, it’s important to understand their implications during automation.

HSTS Strict-Transport-Security: Forces browsers to use HTTPS for a specified duration, preventing MITM attacks. If a site uses HSTS, Playwright will automatically connect via HTTPS after the first visit.
CSP Content-Security-Policy: Mitigates XSS and data injection attacks by specifying allowed sources for content scripts, stylesheets, images, etc.. If a CSP is too strict, your Playwright script might fail to load certain resources or execute injected JavaScript. Debug these failures by checking browser console errors in Playwright’s headed mode.
X-Frame-Options: Prevents clickjacking attacks by controlling whether a page can be rendered in an <frame>, <iframe>, <embed>, or <object>. If a site sets X-Frame-Options: DENY or SAMEORIGIN, Playwright’s frame.goto or page.goto within an iframe context might fail.
X-Content-Type-Options: Prevents MIME-sniffing attacks.
Referrer-Policy: Controls how much referrer information is sent with requests.

Playwright’s Interaction: Playwright honors these security headers just like a real browser. If a CSP blocks a script, it will fail in Playwright too. This is generally a good thing, as it means your automation reflects real-world browser behavior and helps uncover actual security-related issues during testing. When debugging, if you encounter unexpected resource loading failures or script execution issues, the first place to check after network activity should be the browser console for CSP violations.

Common Pitfalls and Troubleshooting

Even with a solid understanding of Playwright headers, you’ll inevitably hit roadblocks. Rotate proxies python

Being prepared for common pitfalls and having a systematic troubleshooting approach can save hours of frustration.

Website Anti-Bot / Anti-Scraping Measures

Many modern websites employ sophisticated techniques to detect and block automated traffic. HTTP headers are often the first line of defense.

Default Playwright Headers: Running Playwright without custom headers is a huge red flag. The default User-Agent HeadlessChrome/X.Y.Z is immediately recognizable. Other subtle header inconsistencies missing Accept-Language, no Sec-Fetch-* headers also stand out.
- Solution: Always set a realistic User-Agent and consider a full set of common browser headers using extraHTTPHeaders for the context.
Header Order/Case Sensitivity: While HTTP headers are generally case-insensitive, some overzealous anti-bot systems might check for specific casing or order. Playwright generally handles this correctly, but if you’re stuck, it’s something to investigate.
Missing or Incorrect Referer: Many sites check the Referer header to ensure a request came from an expected page. If you directly jump to a URL without a proper Referer or with a malformed one, you might be blocked.
- Solution: Use page.gotourl, { headers: { 'Referer': 'https://expected-previous-page.com' } } or ensure your navigation flow naturally sets the Referer.
Dynamic Headers: As discussed, if a site uses anti-CSRF tokens or other dynamic headers, your script must extract and re-inject them.
- Solution: Implement extraction logic for such tokens from HTML, JavaScript, or response headers.
Rate Limiting: Not strictly a header issue, but often related. If your requests are too frequent, a server might respond with 429 Too Many Requests or 503 Service Unavailable.
- Solution: Implement delays page.waitForTimeout or use external proxy networks to distribute traffic.

Debugging Blocked Requests

When you suspect headers are causing a block, here’s a systematic approach:

Start Simple: Try fetching https://httpbin.org/headers with your Playwright setup. This will show you exactly what headers Playwright is sending. Compare these to headers from a real browser using your browser’s developer tools.
page.on'request' and page.on'response': Use these event listeners to log all headers for all requests and responses. This is your most powerful tool for seeing exactly what’s happening. Look for:
- Are all expected headers present?
- Are the header values correct?
- Are there any unexpected headers?
- What is the status code of the response e.g., 403 Forbidden, 429 Too Many Requests, 503 Service Unavailable?
Headless vs. Headed: Sometimes running Playwright in headed mode headless: false and observing the browser’s developer tools Network tab can reveal additional insights or console errors that are not immediately obvious in headless mode.
Try page.route for Debugging: You can use page.route'/*', route => { console.logroute.request.headers. route.continue. }. to log headers for every single request the browser makes, which can sometimes reveal hidden requests that are causing issues.
Isolate the Problem: Comment out sections of your header logic. Remove custom headers one by one. Does it work then? If so, reintroduce them one by one until you find the culprit.
Proxy Logging: If you’re using a proxy e.g., Bright Data, Oxylabs, their dashboards often provide detailed logs of request and response headers, which can be invaluable for debugging.

Incorrect Header Application

Scope Issues: Remember the hierarchy: page.goto headers override context.extraHTTPHeaders, which are typically set globally. If you’re setting headers at the context level but still seeing issues, check if a later page.goto or page.request call is inadvertently overriding them.
Case Sensitivity of Keys: While HTTP header names are technically case-insensitive, JavaScript object keys are not. When you access headers from request.headers or response.headers, Playwright normalizes them to lowercase. So, response.headers is correct, not response.headers. This is a common minor bug.
Overwriting vs. Merging: When using route.continue{ headers: { ...currentHeaders, 'New-Header': 'value' } }, ensure you are correctly spreading ...currentHeaders to merge headers, not just replacing them if you intend to preserve existing ones.

By approaching header-related issues systematically and leveraging Playwright’s powerful debugging tools, you can identify and resolve problems efficiently, leading to more robust and reliable automation scripts.

Remember, web scraping and automation are often a cat-and-mouse game.

Effective header management is one of your strongest weapons.

Frequently Asked Questions

What are HTTP headers in the context of Playwright?

HTTP headers are key-value pairs that contain metadata about a network request or response.

In Playwright, you can send custom headers with your requests or inspect headers received in responses, which is crucial for tasks like web scraping, testing, and authentication.

How do I set a custom User-Agent in Playwright?

You can set a custom User-Agent globally for a BrowserContext using browser.newContext{ userAgent: 'Your Custom User-Agent' } or for specific navigations using page.gotourl, { headers: { 'User-Agent': 'Your Custom User-Agent' } }. It’s vital to use a realistic User-Agent to avoid bot detection.

Can I set headers for all requests in a Playwright session?

Yes, you can set headers for all requests within a BrowserContext by using the extraHTTPHeaders option when creating the context: browser.newContext{ extraHTTPHeaders: { 'Custom-Header': 'value' } }. Burp awesome tls

How do I add headers to a specific `page.goto` request?

You can pass a headers object directly to the page.goto method: await page.goto'https://example.com', { headers: { 'Referer': 'https://another-site.com' } }.. These headers will apply only to that specific navigation.

How do I add headers to Playwright `page.request` API calls e.g., `page.request.post`?

For direct API calls using page.request, headers are passed in the options object: await page.request.post'https://api.example.com/data', { headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer YOUR_TOKEN' }, data: { ... } }..

How can I inspect headers received in a response?

After making a navigation or API call that returns a Response object, you can access its headers using response.headers: const response = await page.goto'https://example.com'. const receivedHeaders = response.headers..

Is it possible to modify request headers dynamically using Playwright?

Yes, you can use page.routeurlPattern, handler to intercept requests and modify their headers on the fly.

Inside the handler, you can call route.continue{ headers: { ...originalHeaders, 'New-Header': 'value' } }..

Can Playwright block certain types of network requests based on headers?

While page.route allows you to abort requests, you’d typically block based on the request URL or resource type rather than headers for efficiency. For example, page.route'/*.{png,jpg}', route => route.abort. blocks images.

How do I handle cookies that are sent as `Set-Cookie` headers?

Playwright handles cookies automatically within a BrowserContext. When a server sends a Set-Cookie header, Playwright stores it and sends it back as a Cookie header on subsequent requests within the same context.

You can also save and load session state, including cookies, using context.storageState.

Why are my Playwright scripts being blocked by websites?

Often, scripts are blocked due to recognizable User-Agent strings, missing common browser headers, inconsistent header profiles, or rapid request rates.

Mimicking a real browser’s header fingerprint and implementing delays can help. Bypass bot detection

What is `extraHTTPHeaders` and when should I use it?

extraHTTPHeaders is an option when creating a BrowserContext browser.newContext. It allows you to set a consistent set of headers that will be sent with every network request originating from that context, ideal for maintaining a consistent browser identity.

How can I debug which headers Playwright is actually sending?

Use page.on'request', request => console.logrequest.url, request.headers. to log outgoing request headers.

Also, navigating to https://httpbin.org/headers with your Playwright script will echo back the headers the server received.

Does Playwright handle `Referer` headers automatically?

Yes, Playwright generally sets the Referer header automatically based on the navigation history, similar to a real browser.

However, you can explicitly override it for specific page.goto calls if needed.

Can I set headers for all browser types Chromium, Firefox, WebKit?

Yes, header manipulation methods like context.extraHTTPHeaders, page.goto{ headers: ... }, and page.request are consistent across all browser engines supported by Playwright.

What is the difference between `page.goto` headers and `page.request` headers?

page.goto headers are applied to the main navigation request for a URL.

page.request allows you to make independent HTTP calls like fetch in the browser without loading the content into the DOM, and its headers apply only to that specific API-style request.

How do I prevent caching in Playwright?

You can instruct Playwright’s browser context to not use caching: browser.newContext{ offline: false } this ensures network calls are always attempted, though the browser might still use internal memory caches for short periods. For more control, you could use page.route to remove Cache-Control headers from outgoing requests or force no-cache.

What are dynamic headers, and how do I manage them?

Dynamic headers are values that change per session or request, such as anti-CSRF tokens, session IDs, or timestamps. Playwright fingerprint

You manage them by scraping the token from a previous response e.g., from HTML or Set-Cookie header and then injecting it into subsequent requests.

Does Playwright handle HTTP/2 and HTTP/3 headers?

Playwright leverages the underlying browser engines, which support HTTP/2 and increasingly HTTP/3. This means headers specific to these protocols like pseudo-headers, or Sec-Fetch-* headers are handled by the browser engine itself, and Playwright can inspect them.

What is the default `User-Agent` for Playwright?

The default User-Agent for Playwright’s Chromium browser typically includes “HeadlessChrome” and a version number, e.g., Mozilla/5.0 Windows NT 10.0. Win64. x64 AppleWebKit/537.36 KHTML, like Gecko HeadlessChrome/108.0.0.0 Safari/537.36. It’s easily detectable by anti-bot systems.

Can I retrieve specific header values from a response?

Yes, after getting the response object, you can access specific header values using the object notation, keeping in mind that Playwright normalizes header names to lowercase: const contentType = response.headers..

Playwright headers