Best web scraping

• Understand the “Why”: Before anything, define your purpose. Are you gathering publicly available data for research, market analysis, or legitimate competitive intelligence? Ensure your objective aligns with ethical data practices and website terms of service.
• Respect robots.txt: This is your first stop. Always check a website’s robots.txt file e.g., example.com/robots.txt. It outlines rules for bots, specifying what areas of the site can and cannot be crawled. Disregarding this is like ignoring a “Do Not Disturb” sign.
• Terms of Service ToS Review: Read the website’s Terms of Service. Many sites explicitly forbid scraping. Adhering to ToS is crucial for legal and ethical compliance.
• Choose Your Tool Wisely:
  • Python Libraries for coders:
    • requests: For making HTTP requests to fetch webpage content.
    • Beautiful Soup: For parsing HTML and XML documents, making it easy to extract data.
    • Scrapy: A powerful, high-level web crawling and scraping framework for larger, more complex projects.
    • Selenium: For scraping dynamic websites that rely heavily on JavaScript, as it automates browser interactions.
  • No-Code Tools for non-coders:
    • Octoparse: A visual web scraping tool for extracting data without coding.
    • ParseHub: Another popular visual tool that handles complex websites.
    • Bright Data’s Web Scraper IDE: Offers a robust, scalable solution.
• Identify Data Points: Pinpoint exactly what data you need e.g., product names, prices, reviews, contact information.
• Inspect Element: Use your browser’s “Inspect Element” feature right-click on the data to understand the HTML structure tags, classes, IDs of the data you want to extract. This is fundamental for writing accurate selectors.
• Write Your Code/Configure Your Tool:
  • Python example simplified:

    import requests
    from bs4 import BeautifulSoup
    
    url = 'https://example.com/data' # Replace with actual URL
    response = requests.geturl
    
    
    soup = BeautifulSoupresponse.text, 'html.parser'
    
    # Example: Find all paragraph tags
    paragraphs = soup.find_all'p'
    for p in paragraphs:
        printp.get_text
    

  • For no-code tools: Follow their graphical interface instructions to select elements.
• Implement Delays and User-Agent Rotation:
  • Delays: Add pauses between requests to avoid overwhelming the server. A delay of 5-10 seconds per request is often a good starting point. This prevents your IP from being blocked.
  • User-Agent Rotation: Rotate your User-Agent header to mimic different browsers, making your scraper less detectable as a bot.
• Handle Dynamic Content: For JavaScript-heavy sites, use tools like Selenium that can render the page before scraping, or explore headless browsers.
• Error Handling: Implement try-except blocks to gracefully handle network errors, missing elements, or changes in website structure.
• Data Storage: Decide how you’ll store the extracted data: CSV, Excel, JSON, or a database SQL, NoSQL.
• Ethical Considerations and Scalability:
  • Respect Server Load: Never launch a scraper that could potentially harm a website’s performance. Distributed Denial of Service DDoS attacks, even accidental ones from scraping, are illegal.
  • Data Usage: Ensure you use the extracted data only for permissible purposes and do not infringe on intellectual property rights.
  • IP Rotation: For large-scale projects, consider using proxy services to rotate IP addresses, further reducing the chances of being blocked. Bright Data offers robust proxy solutions for this.
  • Regular Maintenance: Websites change their structure. Your scraper will need regular updates to keep working.

The Ethical Imperative of Web Scraping

Web scraping, while a powerful tool for data acquisition, carries significant ethical and legal responsibilities. It’s not merely about extracting data. it’s about doing so respectfully and lawfully.

👉 Skip the hassle and get the ready to use 100% working script (Link in the comments section of the YouTube Video) (Latest test 31/05/2025)

Check more on: How to Bypass Cloudflare Turnstile & Cloudflare WAF – Reddit, How to Bypass Cloudflare Turnstile, Cloudflare WAF & reCAPTCHA v3 – Medium, How to Bypass Cloudflare Turnstile, WAF & reCAPTCHA v3 – LinkedIn Article

The very first step before initiating any scraping project must be a thorough understanding of the website’s terms of service and the robots.txt file.

Neglecting these can lead to serious repercussions, from IP blocks to legal action.

Remember, the pursuit of knowledge and data should always be balanced with the principles of fairness, honesty, and respect for others’ digital property.

Just as we wouldn’t take physical items without permission, we should extend that same courtesy to digital information.

Understanding robots.txt and Terms of Service ToS

Before a single line of code is written or a no-code tool is configured, the robots.txt file and the website’s Terms of Service ToS are non-negotiable checks. Get data from web

• robots.txt: This plain text file, typically found at the root of a website e.g., www.example.com/robots.txt, serves as a guideline for web crawlers and scrapers. It specifies which parts of the site are off-limits, which pages can be crawled, and often sets a Crawl-delay directive to prevent overwhelming the server. Respecting robots.txt is a fundamental ethical obligation and a sign of good web citizenship. A study by Incapsula found that nearly 30% of all website traffic is generated by bad bots, underscoring the need for ethical scraping practices.
• Terms of Service ToS: The ToS document outlines the legal agreement between the website owner and its users. Many websites explicitly prohibit scraping, automated data collection, or using their data for commercial purposes without explicit permission. Violating ToS can lead to legal action, cease-and-desist letters, or permanent bans from the website. For instance, LinkedIn’s ToS strictly prohibits scraping, and they have successfully pursued legal action against scrapers. Always err on the side of caution: if the ToS forbids scraping, do not proceed.

The Nuance of Data Ownership and Intellectual Property

Data, especially proprietary data or content that represents significant effort from the website owner, is often protected by intellectual property laws.

• Copyright: The textual content, images, and creative works on a website are typically protected by copyright. Scraping and reusing such content without permission can be a copyright infringement.
• Database Rights: In some jurisdictions like the EU, there are specific database rights that protect the compilation and arrangement of data, even if individual pieces of data are not copyrighted.

Consequences of Unethical Scraping

The repercussions of ignoring ethical and legal boundaries in web scraping can be severe.

• IP Blocking and Blacklisting: Websites can detect and block persistent, aggressive scrapers. This often results in the scraper’s IP address being blacklisted, preventing access to the site entirely.
• Legal Action: Companies are increasingly taking legal action against scrapers who violate their ToS or infringe on their intellectual property. Notable cases include hiQ Labs vs. LinkedIn, where LinkedIn initially blocked hiQ, leading to a protracted legal battle over data access rights.
• Reputational Damage: For businesses or individuals, engaging in unethical scraping can severely damage their reputation, leading to a loss of trust and potential business opportunities.
• Resource Drain on Target Servers: Overly aggressive scraping can overwhelm a website’s servers, leading to slow performance, increased hosting costs, or even downtime. This can be viewed as a malicious act, similar to a Denial of Service DoS attack.

Strategic Choices in Web Scraping Tools

The effectiveness of your web scraping endeavor hinges significantly on the tools you choose.

This decision is primarily dictated by your technical proficiency, the complexity of the website you’re targeting, and the scale of data you need to extract.

For developers, a robust set of Python libraries offers unparalleled flexibility and control.

For those without a coding background, visual, no-code tools provide a much-needed bridge to unlock web data without deep into programming.

It’s about finding the right hammer for the right nail, ensuring efficiency without unnecessary complexity.

Python Libraries: The Developer’s Arsenal

For seasoned developers, Python offers a comprehensive ecosystem of libraries specifically designed for web scraping.

These provide granular control, allowing for highly customized and scalable solutions.

• requests: This library is the cornerstone for making HTTP requests. It simplifies sending GET, POST, PUT, DELETE requests, and handling responses, making it easy to fetch the HTML content of a webpage. It’s like the foundation of your scraping operation. For example, to fetch a page, you’d simply requests.get'https://example.com'. A typical requests operation completes in milliseconds, providing raw HTML quickly.
• Beautiful Soup bs4: Once you have the HTML content from requests, Beautiful Soup comes into play. It’s a Python library for parsing HTML and XML documents. It creates a parse tree that can be navigated, searched, and modified, making it incredibly easy to extract data based on HTML tags, classes, and IDs. It’s excellent for structured data extraction. For instance, to find all links, you’d use soup.find_all'a'. A simple parsing task on a 100KB HTML file often takes less than 500 milliseconds.
• Scrapy: For large-scale, complex scraping projects, Scrapy is a powerful framework. It’s not just a library. it’s a complete ecosystem that handles requests, parsing, data storage, and error handling. Scrapy is highly asynchronous, allowing for multiple requests to be processed concurrently, making it ideal for crawling entire websites. It comes with built-in mechanisms for handling redirects, retries, and offers robust pipeline features for data processing. Many large data companies leverage Scrapy for their data collection needs, with some reporting extracting millions of pages per day.
• Selenium: Modern websites heavily rely on JavaScript to render content dynamically. requests and Beautiful Soup alone cannot execute JavaScript. This is where Selenium shines. It’s a browser automation tool that can control a real browser like Chrome or Firefox programmatically. This allows you to interact with web elements, click buttons, fill forms, and wait for dynamic content to load before scraping. While slower due to launching a full browser instance a page load might take 2-5 seconds, compared to <1 second for requests, it’s essential for JavaScript-rendered content. According to a 2022 survey, over 60% of test automation engineers use Selenium, highlighting its widespread adoption for browser interaction.

No-Code Solutions: Democratizing Data Extraction

For individuals or businesses without programming expertise, no-code web scraping tools offer an accessible way to extract data using visual interfaces. Cloudflare scraping

• Octoparse: This desktop application provides a visual drag-and-drop interface for building scrapers. Users can click on elements they want to extract, and Octoparse intelligently identifies the data pattern. It supports cloud scraping, IP rotation, and scheduling, making it suitable for recurring data needs. Octoparse boasts a user base of over 6 million users globally, many of whom are non-technical.
• ParseHub: Similar to Octoparse, ParseHub is a visual web scraping tool that allows users to create scraping projects by interacting directly with the webpage. It excels at handling complex navigation paths, pagination, and forms. ParseHub also offers powerful features like conditional scraping and robust error handling. They claim their users have scraped over 1 billion pages using their platform.
• Bright Data’s Web Scraper IDE: Bright Data offers a sophisticated Web Scraper IDE Integrated Development Environment that combines the simplicity of visual selection with powerful customization options, allowing users to write custom JavaScript if needed. It’s a hybrid approach that caters to both beginners and advanced users, backed by Bright Data’s extensive proxy network for highly scalable and reliable scraping. Bright Data processes over 100 billion requests daily, demonstrating the robustness of its infrastructure.

Navigating Dynamic Content and Anti-Scraping Measures

This makes direct, static requests insufficient for many scraping tasks.

Overcoming these hurdles requires a deeper understanding of web technologies and the strategic application of advanced tools and methods.

It’s a continuous cat-and-mouse game where scrapers must adapt to new defenses.

Handling JavaScript-Rendered Content

Many contemporary websites load their content dynamically using JavaScript after the initial HTML document is retrieved.

This means that a simple requests.get call will often return an incomplete HTML page, devoid of the data you need.

• Headless Browsers e.g., Selenium, Puppeteer: The most effective solution for JavaScript-rendered content is to use a “headless browser.” A headless browser is a web browser without a graphical user interface GUI that can be controlled programmatically.
  • Selenium: As mentioned earlier, Selenium is widely used. It launches a real browser instance e.g., Chrome via ChromeDriver, Firefox via GeckoDriver in the background, allowing it to execute JavaScript, render the page, and then expose the fully rendered HTML DOM for scraping. You can simulate user interactions like clicking buttons, scrolling, and waiting for elements to appear. While powerful, Selenium is resource-intensive and slower than direct HTTP requests. A typical page load with Selenium might take 2-5 seconds, whereas a simple requests call is in milliseconds.
  • Puppeteer: A Node.js library developed by Google, Puppeteer provides a high-level API to control headless Chrome or Chromium. It’s known for its speed and efficiency in automating browser tasks and is a strong alternative to Selenium for those comfortable with JavaScript.
  • Playwright: A newer library developed by Microsoft, Playwright supports Chromium, Firefox, and WebKit Safari’s rendering engine and offers APIs in multiple languages Python, Node.js, Java, .NET. It’s gaining popularity for its speed, reliability, and ability to handle complex scenarios like parallel execution.
• Identifying API Calls: Sometimes, dynamic content is loaded directly from an API endpoint via AJAX requests. By monitoring network activity in your browser’s developer tools Network tab, you can often identify these API calls. If you can find the direct API URL, you can then make a direct requests call to that API, which is far more efficient than rendering the entire page with a headless browser. This method is faster and less resource-intensive, often retrieving data in tens of milliseconds.

Battling Anti-Scraping Mechanisms

Website owners deploy various techniques to deter or block web scrapers.

Understanding these methods is key to developing resilient scraping solutions.

• IP Blocking and Rate Limiting: The most common defense. If a server detects an unusually high number of requests from a single IP address within a short period, it might temporarily or permanently block that IP.
  • Solution: IP Rotation using proxy services e.g., Bright Data, Oxylabs. These services provide a pool of thousands or millions of residential, datacenter, or mobile IPs, allowing your requests to originate from different locations, making it harder for the target website to identify and block you. Bright Data offers residential proxies with a success rate often exceeding 99% for many targets.
  • Solution: Adding Delays and Randomization. Implement time.sleep in your code between requests to simulate human browsing patterns. Varying the delay e.g., between 5 to 15 seconds makes it even less predictable.
• User-Agent and Header Checks: Websites often inspect the User-Agent header to determine if a request is coming from a legitimate browser or a known bot. If it’s a generic Python requests User-Agent, it might be flagged.
  • Solution: Rotate User-Agents. Maintain a list of legitimate User-Agent strings e.g., from Chrome, Firefox, Safari on different OS and randomly select one for each request. You can find comprehensive lists of User-Agents online.
• CAPTCHAs Completely Automated Public Turing test to tell Computers and Humans Apart: These are designed to distinguish between human users and bots. They range from simple image recognition reCAPTCHA v2 to invisible background checks reCAPTCHA v3.
  • Solution: CAPTCHA Solving Services. Services like 2Captcha or Anti-Captcha offer APIs to integrate human CAPTCHA solving into your scraping workflow. You send them the CAPTCHA, a human solves it, and returns the token. This adds cost and latency but can be effective.
  • Solution: For some simple CAPTCHAs, machine learning models like OCR might be trained, but modern CAPTCHAs are specifically designed to defeat automated systems.
• Honeypot Traps: These are invisible links or elements on a webpage specifically designed to trap bots. When a bot clicks on such an element which a human wouldn’t see, it’s identified as a bot and its IP is blocked.
  • Solution: Careful Element Selection. When writing your parsing logic, ensure you are only selecting visible, user-facing elements that are relevant to your data extraction. Avoid selecting all <a> tags indiscriminately.
• Website Structure Changes: Websites frequently update their layouts, HTML class names, and IDs. This can break your scraper, as the selectors you defined are no longer valid.
  • Solution: Robust Selectors and Regular Maintenance. Design your selectors to be as resilient as possible e.g., using parent-child relationships, or unique attributes instead of volatile class names. Implement monitoring to detect when your scraper breaks and be prepared for regular maintenance and updates. Companies often dedicate 20-30% of their scraping budget to maintenance and anti-bot evasion.

Data Storage, Cleaning, and Management

Extracting data is only the first step.

The true value comes from how you store, clean, and manage that data.

Raw scraped data is often messy, inconsistent, and requires significant processing before it can be used for analysis or integration into other systems. Api to scrape data from website

Effective data management ensures data integrity, accessibility, and utility, transforming a collection of raw bits into actionable intelligence.

This process is crucial for any data-driven initiative, ensuring that the effort spent on scraping translates into tangible, reliable insights.

Choosing the Right Storage Format

The decision of where and how to store your scraped data depends on its volume, structure, and your subsequent analytical or operational needs.

• CSV Comma Separated Values:
  • Pros: Simple, universal, human-readable, easily opened in spreadsheet software Excel, Google Sheets. Excellent for smaller datasets and tabular data.
  • Cons: Lacks structural complexity no nested data, difficult to manage large volumes, no built-in schema enforcement, can be prone to delimiter issues if data contains commas.
  • Use Case: Ideal for quick exports of structured tables e.g., product lists, simple contact information. Many web scraping tools offer direct CSV export.
• JSON JavaScript Object Notation:
  • Pros: Flexible, hierarchical structure, human-readable, widely used in web APIs, easily parsed by most programming languages. Supports nested data, making it suitable for more complex data structures.
  • Cons: Less suitable for direct spreadsheet viewing without conversion, can be challenging to query without programming.
  • Use Case: Perfect for semi-structured data, nested data e.g., product details with multiple attributes, reviews with authors and dates, or when interacting with APIs. Most modern scraping frameworks can output JSON.
• SQL Databases e.g., PostgreSQL, MySQL, SQLite:
  • Pros: Structured, robust, supports complex queries SQL, ensures data integrity with schemas, excellent for large datasets, transactional support.
  • Cons: Requires schema definition upfront, can be more complex to set up and manage, less flexible for rapidly changing data structures.
  • Use Case: Best for large-scale, structured datasets where data integrity and complex querying are paramount e.g., e-commerce product catalogs, long-term historical price data. SQLite is great for local, file-based databases. PostgreSQL and MySQL for production servers.
• NoSQL Databases e.g., MongoDB, Cassandra:
  • Pros: Flexible schema document-oriented, excellent for unstructured or semi-structured data, highly scalable for very large datasets, high performance for specific access patterns.
  • Cons: Less emphasis on data integrity no strict schema, different querying paradigms not SQL, can be harder to manage for beginners.
  • Use Case: Ideal for large volumes of rapidly changing or unstructured data, such as real-time news feeds, social media data, or when the data structure isn’t fixed. MongoDB is particularly popular for document-based storage.

Essential Data Cleaning Techniques

Raw scraped data is rarely pristine. It often contains inconsistencies, missing values, duplicates, and irrelevant information. Data cleaning is a critical step to ensure data quality and usability. Studies show that data scientists spend up to 80% of their time on data preparation and cleaning.

• Handling Missing Values:
  • Identify: Determine which fields have missing data.
  • Impute: Fill missing values with a placeholder, the mean/median/mode of the column, or use more advanced imputation techniques.
  • Remove: If missing values are extensive or critical, consider removing rows or columns.
• Removing Duplicates:
  • Identify and remove identical records. This is crucial when scraping multiple pages or over time. Using unique identifiers like product IDs or URLs is key. Python’s Pandas library offers df.drop_duplicates for efficient removal.
• Standardizing Formats:
  • Dates: Convert various date formats e.g., “01/01/2023”, “Jan 1, 2023” into a consistent format e.g., “YYYY-MM-DD”.
  • Currencies: Convert different currency symbols $, €, £ to a common format or separate them into distinct columns.
  • Text Case: Standardize text to lowercase or uppercase for consistency in search and analysis e.g., “Apple” vs. “apple”.
• Data Type Conversion:
  • Ensure numerical data prices, ratings is stored as numbers, not strings. This allows for mathematical operations.
  • Convert boolean strings “True”, “False” to actual boolean types.
• Removing Irrelevant Characters/Whitespace:
  • Strip leading/trailing whitespace from text fields.
  • Remove special characters, HTML tags, or unnecessary newline characters \n. Regular expressions are powerful for this.
• Error Correction:
  • Correct typos or inconsistencies in categorical data e.g., “USA”, “U.S.A.”, “United States” should all be standardized to “United States”.

Version Control and Data Pipelines

For ongoing scraping projects, robust data management extends to version control and automated pipelines.

• Version Control for Code: Use Git and GitHub/GitLab to manage your scraping scripts. This tracks changes, allows collaboration, and makes it easy to revert to previous versions if a new change breaks the scraper.
• Data Pipelines: Automate the entire process from scraping to storage and cleaning.
  • ETL Extract, Transform, Load: A common data pipeline paradigm.
    • Extract: Scrape the raw data.
    • Transform: Clean, validate, and structure the data.
    • Load: Store the cleaned data into your chosen database or storage format.
  • Tools like Apache Airflow, Prefect, or simple cron jobs for scheduled scripts can orchestrate these pipelines, ensuring data is collected, processed, and available regularly. For instance, a daily scrape of e-commerce prices might run on a cron job, feeding into an ETL pipeline that cleans and loads the data into a PostgreSQL database for analysis.
  • According to a survey by Cloud Native Computing Foundation, over 70% of organizations are using or experimenting with data pipeline tools to manage their data workflows.

Ethical Data Usage and Legal Compliance

The act of web scraping, while technically feasible, is fraught with ethical and legal complexities.

It’s paramount to understand that just because data is publicly accessible does not mean it’s free for unregulated use, especially for commercial purposes.

Operating within the bounds of law and ethics is not merely a formality but a necessity to avoid legal repercussions, maintain a positive reputation, and contribute responsibly to the digital ecosystem.

Ignoring these aspects can lead to costly lawsuits, reputational damage, and even criminal charges in some extreme cases.

Adhering to Laws and Regulations

• Copyright Law: The content displayed on websites text, images, videos, design elements is almost always protected by copyright. Scraping copyrighted material and then reproducing, distributing, or adapting it without permission is a direct infringement. For example, scraping articles from a news website and republishing them on another site would be a clear copyright violation. In the U.S., the Digital Millennium Copyright Act DMCA can be invoked against unauthorized scraping.
• Database Rights: In certain regions, particularly the European Union, specific sui generis database rights protect the investment made in collecting and organizing data, even if the individual data points are not copyrighted. This means that extracting substantial parts of a database without permission can be illegal. The sui generis right prevents extraction and/or re-utilization of the whole or a substantial part of the contents of a database.
• Terms of Service ToS and End User License Agreements EULA: As discussed, these are legally binding contracts between the website owner and the user. Violating a website’s ToS by scraping can be considered a breach of contract, leading to legal action. For instance, in the LinkedIn vs. hiQ Labs case, a central argument was whether LinkedIn’s ToS prohibited hiQ’s scraping activities. While the initial ruling favored hiQ, the legal battle continues, highlighting the complexities.
• Data Protection Regulations e.g., GDPR, CCPA: If the data you are scraping contains “personal data” any information relating to an identified or identifiable natural person, then stringent data protection laws come into play.
  • GDPR General Data Protection Regulation: Applicable if you are scraping data from individuals in the EU or if your organization is based in the EU. GDPR mandates strict rules for collecting, processing, and storing personal data, including requirements for consent, data minimization, and the “right to be forgotten.” Fines for GDPR violations can be substantial, up to €20 million or 4% of global annual turnover, whichever is higher.
  • CCPA California Consumer Privacy Act: Grants California consumers specific rights regarding their personal information. Similar laws are emerging in other U.S. states.
  • Implication for Scraping: Scraping personal data e.g., names, email addresses, phone numbers, online identifiers without a clear legal basis like consent or legitimate interest is highly problematic under these regulations. Data minimization is key: only collect what is absolutely necessary.
• Computer Fraud and Abuse Act CFAA: In the U.S., the CFAA prohibits unauthorized access to computers. While primarily aimed at hacking, some interpretations have sought to apply it to web scraping that bypasses technical access restrictions. Cases like Facebook vs. Power Ventures have shown the potential for CFAA to be invoked against scrapers.

Ethical Considerations Beyond Legality

While legal compliance is a baseline, ethical considerations often extend beyond what is strictly prohibited by law. Java web scraping

• Respect for Website Resources: Even if not explicitly forbidden, aggressive scraping that floods a server with requests and degrades website performance is unethical. It can harm the target website’s user experience and financial viability. This is why incorporating delays and respecting Crawl-delay directives in robots.txt is crucial.
• Transparency and Attribution: If you are using scraped data publicly, consider being transparent about its source and, where appropriate, provide attribution to the original website. This fosters good relationships and acknowledges the effort of the original content creators.
• Data Minimization: Collect only the data you genuinely need. Avoid scraping entire websites indiscriminately if your objective only requires specific pieces of information. This reduces storage burden and minimizes the risk associated with handling unnecessary data, especially personal data.
• Impact on the Digital Ecosystem: Consider the broader implications of your scraping activities. Is it contributing positively to data availability and research, or is it potentially undermining business models, intellectual property, or user privacy? A responsible data professional always considers the wider impact of their actions.

Best Practices for Responsible Scraping

To navigate the ethical and legal minefield, adopt these best practices:

• Always Check robots.txt: This is your first and most fundamental ethical check.
• Review ToS Thoroughly: If the ToS prohibits scraping, respect it. If there’s ambiguity, seek legal advice.
• Scrape Responsibly:
  • Rate Limit Your Requests: Implement delays between requests e.g., 5-15 seconds per request.
  • Use Proxies Judiciously: While useful for scalability, don’t use proxies to mask unethical behavior.
  • Identify Your Scraper if appropriate: Set a custom User-Agent that clearly identifies your scraper, allowing website owners to contact you if there’s an issue e.g., MyScraper/1.0 contact: [email protected].
• Prioritize Publicly Available Data: Focus on data clearly intended for public consumption and where privacy concerns are minimal.
• Avoid Personal Data unless Legally Justified: Be extremely cautious when scraping any data that could be identified as personal. If you must, ensure you have a legitimate legal basis and adhere to all relevant data protection laws GDPR, CCPA, etc.. An overwhelming 88% of consumers are concerned about data privacy, according to a recent Pew Research study, highlighting the public’s sensitivity.
• Secure Stored Data: If you collect any data, especially personal data, ensure it is stored securely and protected from unauthorized access.

Advanced Scraping Techniques and Tools

As websites become more sophisticated, basic scraping techniques often fall short.

To extract data from complex, dynamic, or heavily protected sites, you need to employ advanced strategies and leverage specialized tools.

This section dives into methods for overcoming challenging scenarios, pushing beyond the basic requests and Beautiful Soup setup.

It’s about enhancing resilience, efficiency, and scale in your scraping operations.

Proxy Networks for Scalability and Anonymity

When performing large-scale scraping or targeting sites with aggressive IP blocking, a single IP address will quickly be identified and blacklisted. Proxy networks are essential for overcoming this.

• What are Proxies?: Proxies act as intermediaries between your scraper and the target website. Your request goes through the proxy server, which then forwards it to the website. The website sees the proxy’s IP address, not yours.
• Types of Proxies:
  • Datacenter Proxies: IPs originating from data centers. They are fast and cheap but easily detected by sophisticated anti-bot systems. Best for non-sensitive, large-volume scraping of less protected sites.
  • Residential Proxies: IPs assigned by Internet Service Providers ISPs to real home users. They are much harder to detect because they appear as legitimate user traffic. Ideal for scraping highly protected or JavaScript-heavy sites. Bright Data offers a vast network of residential proxies, with millions of IPs globally. Reports indicate success rates for residential proxies can be over 99% on challenging targets.
  • Mobile Proxies: IPs from mobile carriers 3G/4G/5G. These are the most expensive but offer the highest trust level as mobile IPs are frequently rotated by carriers and are less likely to be flagged. Best for extremely sensitive targets.
• Proxy Rotation: Simply using one proxy isn’t enough. Effective scraping involves rotating through a large pool of proxies, often on a per-request basis or after a certain number of requests, to distribute the load and further mimic human behavior. Services like Bright Data handle this rotation automatically.
• Geolocation Targeting: Proxies also allow you to specify the geographic location of the IP, which is crucial for scraping region-specific content or ensuring compliance with local data regulations.

Distributed Scraping and Cloud Infrastructure

For truly massive scraping projects, a single machine won’t suffice.

Distributed scraping involves running your scraper across multiple machines, often leveraging cloud computing services.

• Benefits:
  • Scalability: Process vast amounts of data by scaling up computing resources as needed.
  • Speed: Scrape many pages concurrently across different machines.
  • Resilience: If one machine fails, others can continue the task.
• Implementation:
  • Cloud Platforms: AWS Amazon Web Services, Google Cloud Platform GCP, and Microsoft Azure offer virtual machines EC2, Compute Engine, Virtual Machines that can host your scraping scripts.
  • Containerization Docker: Package your scraper and its dependencies into Docker containers. This ensures consistency across different machines and simplifies deployment.
  • Orchestration Kubernetes: For managing many containers, Kubernetes automates deployment, scaling, and management of containerized applications.
  • Message Queues e.g., RabbitMQ, Apache Kafka: Use message queues to distribute tasks e.g., URLs to scrape among worker machines and collect results, ensuring efficient communication and fault tolerance. A typical distributed scraping setup might involve hundreds or thousands of concurrent requests, processing data from millions of pages monthly.

CAPTCHA Solving Services and AI

CAPTCHAs are a significant hurdle for automated scrapers.

Ai web scraping python

While manual solving is an option, services and AI can streamline the process.

• Third-Party CAPTCHA Solving Services: Services like 2Captcha, Anti-Captcha, or DeathByCaptcha provide APIs to integrate human-powered CAPTCHA solving into your workflow. Your scraper sends the CAPTCHA image/data to the service, human workers solve it, and the solution is returned. This adds latency and cost but is highly effective for complex CAPTCHAs. Costs typically range from $0.50 to $2 per 1000 CAPTCHAs.
• Machine Learning for CAPTCHA Bypass: For simpler, older CAPTCHA types, machine learning models e.g., using OpenCV for image processing, or TensorFlow/PyTorch for neural networks can sometimes be trained to solve them. However, modern CAPTCHAs like reCAPTCHA v3 which uses behavioral analysis are designed to be difficult for automated systems to bypass, requiring sophisticated AI or real human interaction.

Web Scraping Frameworks with Built-in Advanced Features

Certain frameworks offer features that abstract away much of the complexity of advanced scraping.

• Scrapy Revisited: Beyond basic scraping, Scrapy provides:
  • AutoThrottling: Adjusts crawl speed based on server load, respecting server resources.
  • Request Prioritization: Allows you to prioritize certain types of requests.
  • Middleware: Custom logic can be inserted into the request/response cycle for tasks like proxy rotation, User-Agent rotation, cookie handling, and IP blocking detection.
  • Item Pipelines: Process and store scraped data, handling cleaning, validation, and storage.
  • Scrapy’s robust architecture allows it to handle projects involving millions of URLs with high efficiency.
• Bright Data’s Web Scraper IDE: As mentioned, this tool bridges the gap between no-code and code-based solutions. It allows users to visually select data points while also providing the option to inject custom JavaScript for complex interactions. Its integration with Bright Data’s proxy network makes it highly scalable and resilient against anti-bot measures, making it a powerful choice for both beginners and advanced users.

Monitoring, Maintenance, and Scalability

Web scraping is rarely a “set it and forget it” operation.

Websites are dynamic entities, constantly changing their structure, content, and anti-scraping defenses.

Therefore, continuous monitoring, regular maintenance, and strategic scalability are crucial for ensuring the long-term effectiveness and reliability of your scraping operations.

Without these practices, even the most expertly built scraper will eventually fail, leading to stale data and wasted effort.

Continuous Monitoring for Breakages and Data Quality

A scraper can break for various reasons, and it’s essential to detect these issues quickly.

• Website Structure Changes: The most common cause. Websites frequently update their HTML class names, IDs, nesting, breaking the selectors in your scraper.
• Anti-Scraping Updates: Websites may deploy new CAPTCHAs, stricter rate limits, or advanced bot detection mechanisms.
• Server-Side Issues: The target website might be down, experiencing slow performance, or returning unexpected error codes e.g., 404 Not Found, 500 Internal Server Error.
• Network Issues: Your own network, proxy network, or internet connectivity can cause failures.
• Monitoring Tools and Strategies:
  • Automated Error Reporting: Implement try-except blocks in your code to catch errors and log them. Send error notifications via email, Slack, or dedicated monitoring services e.g., Sentry, New Relic.
  • Data Validation Checks: After a scrape, perform quick checks on the extracted data. Are there unexpected missing values? Is the number of records significantly lower than usual? Are data types correct?
  • Scheduled Health Checks: Run a small “health check” scrape frequently e.g., hourly against a few critical pages to ensure the scraper is still functioning.
  • Dashboarding: For larger operations, visualize scraping success rates, error rates, and data volume over time using tools like Grafana, Kibana, or even simple custom dashboards. This provides an at-a-glance overview of your scraping health. A well-monitored scraping pipeline can often detect issues within minutes of them occurring, significantly reducing data downtime.

Regular Maintenance and Updates

Responding to monitoring alerts requires proactive maintenance.

• Selector Updates: When a website’s HTML structure changes, update your CSS selectors or XPath expressions to match the new layout. This is the most frequent maintenance task.
• Anti-Scraping Bypass Adaptations: If a website deploys new anti-bot measures, you’ll need to research and implement new strategies e.g., integrating a new CAPTCHA solving service, adjusting proxy rotation logic, or fine-tuning delays.
• Dependency Updates: Keep your Python libraries, browser drivers for Selenium, and other software dependencies up-to-date to ensure compatibility and benefit from bug fixes and performance improvements.
• Code Refactoring and Optimization: Periodically review your scraper’s code for efficiency, readability, and maintainability. Optimize for speed and resource usage.
• Documentation: Maintain clear documentation of your scraping logic, data fields, and any specific website-related quirks. This is invaluable for troubleshooting and for new team members. Many scraping teams report dedicating 15-25% of their time to maintenance tasks.

Scaling Your Scraping Operations

As your data needs grow, your scraping infrastructure must scale accordingly.

• Horizontal Scaling Distributed Scraping:
  • Instead of running one powerful scraper, run many smaller scrapers concurrently across multiple machines.
  • Cloud Services: Utilize cloud platforms AWS, GCP, Azure to spin up and down virtual machines as needed.
  • Containerization Docker & Kubernetes: Package your scrapers as Docker containers and orchestrate them with Kubernetes. This allows for seamless deployment, auto-scaling, and load balancing across a cluster of machines.
  • Message Queues: Use message queues RabbitMQ, Kafka, SQS to manage URLs to be scraped and results to be processed. This decouples components and makes the system more robust and scalable. For example, a single producer can add URLs to a queue, and multiple scraper workers can pick them up, process them, and then add results to another queue for processing by a data storage worker.
• Proxy Management: For large-scale operations, robust proxy management is non-negotiable.
  • Dedicated Proxy Providers: Partner with commercial proxy providers Bright Data, Oxylabs that offer large pools of high-quality proxies, automatic rotation, and advanced features like geo-targeting. They handle the complexity of maintaining millions of IPs.
  • IP Ban Management: Implement logic to detect IP bans and automatically switch to a new proxy from your pool.
• Data Storage Scalability:
  • Ensure your data storage solution database, data lake can handle the increasing volume of data. Cloud databases AWS RDS, Google Cloud SQL offer managed scaling. For massive volumes of unstructured data, consider data lakes AWS S3, Google Cloud Storage combined with big data processing frameworks Apache Spark.
  • A typical large-scale scraping operation might process hundreds of millions of pages per month, generating terabytes of data.

Best Practices and Ethical Web Scraping

Beyond technical execution, the “best” web scraping hinges on a foundation of ethical conduct and adherence to best practices. This isn’t just about avoiding legal trouble. Url scraping

It’s about being a responsible digital citizen and ensuring that your data collection efforts are sustainable and respectful of the broader internet ecosystem.

Neglecting these principles undermines the integrity of your work and can harm the very platforms you rely on for data.

Prioritizing Ethical Conduct

The line between acceptable and unacceptable scraping is often blurry, making ethical considerations paramount.

• Respect for robots.txt: This file is a clear directive from the website owner regarding what parts of their site should not be crawled. Ignoring it is akin to trespassing. Always check it first.
• Adherence to Terms of Service ToS: The ToS outlines the legal agreement for using a website. If it prohibits scraping, respect that. Attempting to bypass ToS can lead to legal action, as seen in numerous high-profile cases.
• Do Not Cause Harm: Never develop or deploy a scraper that could potentially disrupt a website’s operations, cause server overload, or incur excessive costs for the target site. This means implementing polite scraping practices like rate limiting and staggered requests. A sudden surge of requests from a scraper can sometimes be misinterpreted as a Denial of Service DoS attack, even if unintended.
• Data Minimization: Collect only the data you genuinely need for your stated purpose. Avoid hoarding vast amounts of irrelevant data, especially if it contains personal identifiable information PII.
• Transparency Where Possible: In some contexts, it might be beneficial to identify your scraper with a unique User-Agent string e.g., MyResearchScraper/1.0 contact: [email protected]. This allows website administrators to contact you if they have concerns, fostering goodwill.

Implementing Polite Scraping Practices

Beyond ethical principles, specific technical practices can make your scraper a “good neighbor.”

• Rate Limiting and Delays: This is the most crucial polite practice. Introduce delays between requests to avoid overwhelming the server.
  • Fixed Delays: A simple time.sleepX e.g., 5-10 seconds between requests.
  • Random Delays: Vary the delay within a range e.g., time.sleeprandom.uniform5, 15 to make your request patterns less predictable and less like a bot.
  • Respect Crawl-delay: If robots.txt specifies a Crawl-delay e.g., Crawl-delay: 10, respect this value.
  • Statistics show that websites experience significantly fewer IP blocks from scrapers that implement delays of at least 5 seconds between requests.
• User-Agent Rotation: As discussed, rotate through a pool of legitimate User-Agent strings to mimic different browsers. This helps your scraper blend in with regular user traffic.
• Handling HTTP Status Codes Gracefully:
  • 200 OK: Success.
  • 403 Forbidden: Your request was understood, but the server refuses to fulfill it often due to anti-bot measures.
  • 404 Not Found: The requested resource doesn’t exist.
  • 429 Too Many Requests: You’ve sent too many requests in a given amount of time. Implement longer delays or switch proxies.
  • 5xx Server Errors: Issues on the website’s server. Implement retry logic with exponential backoff for these errors.
• Using Sessions for Cookie Management: For sites requiring login or session persistence, use a requests.Session object. This handles cookies automatically, maintaining your session across multiple requests.

Data Security and Privacy

If your scraping activities involve personal data, data security and privacy become non-negotiable.

• Encryption: Encrypt data both in transit using HTTPS for requests and at rest when storing it on disks or databases.
• Access Control: Restrict access to scraped data to only authorized personnel. Implement strong authentication and role-based access control.
• Anonymization/Pseudonymization: If you scrape personal data for analytical purposes, consider anonymizing or pseudonymizing it e.g., hashing email addresses to protect privacy while retaining analytical utility.
• Compliance with Data Protection Laws: If you scrape personal data from individuals in regions with strict data privacy laws like GDPR in Europe, CCPA in California, ensure full compliance. This often involves understanding legal bases for processing, data subject rights, and data retention policies. A Deloitte survey found that 75% of organizations consider data privacy a critical component of their business strategy.

Scalability and Reliability Considerations

For any serious scraping operation, building for scale and reliability from the outset saves immense headaches down the line.

• Modular Design: Break down your scraper into small, independent modules e.g., separate modules for fetching, parsing, storing, error handling. This makes debugging, maintenance, and scaling much easier.
• Error Handling and Logging: Implement robust error handling using try-except blocks to gracefully manage exceptions. Comprehensive logging e.g., using Python’s logging module helps in diagnosing issues and monitoring scraper health.
• Version Control: Use Git for your scraping code. This allows you to track changes, collaborate effectively, and revert to stable versions if a new update introduces bugs.
• Automated Testing: Write unit tests for your parsing logic and integration tests for key parts of your scraper. Automated tests can quickly identify when a website change breaks your scraper.
• Cloud Infrastructure: As your needs grow, leverage cloud platforms AWS, GCP, Azure for scalable compute power virtual machines, serverless functions and storage databases, object storage like S3.
• Proxy Solutions: For large-scale or robust scraping, integrate with a reputable proxy service provider e.g., Bright Data. They manage large pools of IPs, rotation, and geo-targeting, offloading significant operational burden from you.

Maximizing Value from Scraped Data

The ultimate goal of web scraping isn’t just to collect data, but to transform it into actionable intelligence.

Raw scraped data, no matter how vast, holds little inherent value until it’s properly processed, analyzed, and integrated into decision-making processes.

This final stage involves meticulous data post-processing, insightful analysis, and strategic application to derive business value, research insights, or competitive advantage.

It’s about moving from data collection to data utilization. Web page scraping

Data Validation and Enrichment

Once data is scraped and initially cleaned, further validation and enrichment can significantly enhance its quality and utility.

• Validation Rules: Define rules to check the integrity and accuracy of your data.
  • Completeness: Are all required fields present?
  • Format: Do values adhere to expected formats e.g., email addresses, phone numbers?
  • Range: Are numerical values within plausible ranges e.g., prices not negative?
  • Consistency: Are categorical values consistent across records e.g., “In Stock” vs. “in-stock”?
  • Tools like pandera for Python Pandas DataFrames allow you to define and enforce data validation schemas.
• Data Enrichment: Enhance your scraped data by combining it with external datasets or services.
  • Geo-coding: If you scrape addresses, use a geo-coding API e.g., Google Maps API, OpenStreetMap Nominatim to convert them into latitude and longitude coordinates, enabling spatial analysis.
  • Company Data: If you scrape company names, enrich them with public financial data, industry classifications, or employee counts from business directories or APIs.
  • Sentiment Analysis: For scraped reviews or social media comments, apply natural language processing NLP techniques to determine the sentiment positive, negative, neutral, providing valuable insights into customer perception.
  • Currency Conversion: If prices are scraped in multiple currencies, convert them to a single base currency using up-to-date exchange rates from an API.
  • A study by Forrester found that organizations that excel at data enrichment see a 25% increase in revenue.

Analytics and Visualization

Once the data is clean and enriched, the real work of uncovering insights begins.

• Statistical Analysis: Apply statistical methods to identify trends, correlations, and anomalies.
  • Descriptive Statistics: Calculate means, medians, standard deviations to summarize your data e.g., average product price, distribution of ratings.
  • Inferential Statistics: Test hypotheses and make predictions e.g., compare prices across different e-commerce platforms, analyze the impact of review sentiment on sales.
• Data Mining and Machine Learning:
  • Clustering: Group similar items together e.g., identify clusters of similar products based on their scraped features.
  • Classification: Predict categories e.g., classify news articles by topic.
  • Regression: Predict numerical values e.g., predict future stock prices based on historical scraped data, though this requires extreme caution and expertise.
• Data Visualization: Present your findings clearly and compellingly using charts, graphs, and dashboards.
  • Tools: Tableau, Power BI, Google Data Studio, Matplotlib/Seaborn Python, ggplot2 R.
  • Examples:
    • Price Trends: Line charts showing price fluctuations of a product over time.
    • Market Share: Bar charts showing the presence of different brands on a marketplace.
    • Review Sentiment: Pie charts or bar charts illustrating the proportion of positive, negative, and neutral reviews.
    • Geographic Analysis: Heatmaps showing product availability or pricing variations across different regions.
  • 90% of information transmitted to the brain is visual, and visuals are processed 60,000 times faster in the brain than text, underscoring the power of visualization.

Integration and Application

The ultimate goal is to integrate scraped data into existing systems and leverage it for strategic applications.

• Business Intelligence BI Systems: Integrate clean, analyzed scraped data into your BI dashboards to provide real-time insights for decision-makers. This can include competitive pricing intelligence, market trend analysis, or product performance tracking.
• Marketing and Sales:
  • Lead Generation: Scrape public directories for potential client contact information always adhering to privacy laws.
  • Competitor Analysis: Monitor competitor pricing, product launches, and promotional strategies.
  • Sentiment Monitoring: Track public opinion about your brand or products across various platforms.
• E-commerce and Retail:
  • Dynamic Pricing: Adjust your product prices in real-time based on competitor pricing scraped from other sites.
  • Product Assortment Optimization: Identify popular products, gaps in your inventory, or emerging trends based on competitor listings.
  • Review Aggregation: Collect and analyze customer reviews from multiple platforms to improve product offerings.
• Research and Development:
  • Academic Research: Gather data for linguistic analysis, social science studies, or economic modeling.
  • Patent Analysis: Scrape patent databases for competitive intelligence or innovation trends.
• Risk Management:
  • Fraud Detection: Scrape public records to identify patterns indicative of fraudulent activities.
  • Brand Monitoring: Track mentions of your brand to detect potential reputational risks or intellectual property infringements.
  • Companies that effectively integrate scraped data into their operations report a 10-15% increase in operational efficiency and significant improvements in decision-making speed.

Frequently Asked Questions

What is web scraping?

Web scraping is the automated process of extracting data from websites.

It involves programmatically sending requests to web servers, retrieving HTML content, and then parsing that content to extract specific information, often to be stored in a structured format like a database or spreadsheet.

Is web scraping legal?

The legality of web scraping is complex and depends on several factors, including the website’s terms of service, the type of data being scraped especially if it’s personal data, the jurisdiction you’re operating in, and how you intend to use the data. Always check robots.txt and the website’s ToS.

Scraping personal data is generally problematic under GDPR and CCPA without a legal basis.

Can I scrape any website?

No, you cannot scrape any website without restrictions.

Many websites explicitly prohibit scraping in their Terms of Service ToS or through their robots.txt file.

Additionally, some websites employ anti-bot measures to prevent scraping. Always respect these directives. Api get

What are the best programming languages for web scraping?

Python is widely considered the best programming language for web scraping due to its rich ecosystem of libraries like requests, Beautiful Soup, Scrapy, and Selenium. Other languages like Node.js with Puppeteer or Playwright, Ruby, and PHP also have scraping capabilities but are less dominant.

What is robots.txt and why is it important?

robots.txt is a file that webmasters use to communicate with web crawlers and scrapers, indicating which parts of their website they prefer not to be accessed or crawled.

It’s important to respect robots.txt as a matter of ethical conduct and to avoid legal issues, as ignoring it can be seen as an unauthorized access.

What are Terms of Service ToS and how do they affect scraping?

Terms of Service ToS are legal agreements between a website owner and its users.

Many ToS documents explicitly prohibit automated data collection or scraping.

Violating a website’s ToS by scraping can be considered a breach of contract, leading to legal action.

What are anti-scraping measures?

Anti-scraping measures are techniques websites use to deter or block automated scrapers.

These can include IP blocking, rate limiting, CAPTCHAs, User-Agent string checks, honeypot traps, and complex JavaScript rendering that makes data extraction difficult for simple scrapers.

How do I handle dynamic content JavaScript-rendered when scraping?

To handle dynamic content loaded by JavaScript, you typically need to use a headless browser like Selenium, Puppeteer, or Playwright.

These tools can automate a real browser, execute JavaScript, and wait for content to load before extracting the data from the fully rendered page. Web scraping cloudflare

What are proxies and why are they used in web scraping?

Proxies are intermediary servers that forward your web requests.

They are used in web scraping to hide your real IP address, rotate IP addresses to avoid detection and blocking, and to mimic requests from different geographic locations.

Residential proxies are often preferred for their higher success rates on challenging sites.

What is the difference between datacenter and residential proxies?

Datacenter proxies originate from commercial data centers and are generally faster and cheaper but easier to detect.

Residential proxies are assigned by Internet Service Providers ISPs to real home users, making them appear as legitimate user traffic and much harder to detect by anti-bot systems.

What is a User-Agent and why should I rotate it?

A User-Agent is a string sent with every HTTP request that identifies the browser or application making the request.

Rotating User-Agents mimicking different legitimate browsers like Chrome, Firefox, Safari makes your scraper appear less like a bot and can help avoid detection and blocking.

How often should I scrape a website?

The frequency of scraping depends entirely on the website’s terms of service, the volatility of the data, and the server load you are imposing.

For ethical reasons, you should scrape as infrequently as possible.

For critical data, you might scrape daily, but for static data, weekly or monthly might suffice. Always implement delays between requests. Scrape data from website python

What are the common data storage formats for scraped data?

Common data storage formats include CSV for simple tabular data, JSON for semi-structured or nested data, and databases like SQL PostgreSQL, MySQL for structured, large-scale data, or NoSQL MongoDB for flexible, large-volume data.

How do I clean scraped data?

Data cleaning involves handling missing values, removing duplicates, standardizing formats dates, currencies, text case, converting data types, removing irrelevant characters or whitespace, and correcting errors. This process ensures data quality and usability.

What is ethical scraping?

Ethical scraping means collecting data responsibly and respectfully.

This involves respecting robots.txt and ToS, implementing rate limiting and delays, avoiding personal data unless legally permissible, not causing harm to the target website, and being transparent about your scraping activities where appropriate.

Can web scraping lead to legal issues?

Yes, web scraping can lead to legal issues such as copyright infringement, breach of contract for violating ToS, and violations of data protection laws like GDPR or CCPA if personal data is involved.

In some cases, it could also be considered unauthorized computer access under laws like the CFAA.

What is a headless browser?

A headless browser is a web browser without a graphical user interface GUI that can be controlled programmatically.

It’s used in web scraping to execute JavaScript on a webpage, render dynamic content, and simulate user interactions, making it effective for scraping modern, JavaScript-heavy websites.

What is a CAPTCHA and how can scrapers bypass them?

A CAPTCHA Completely Automated Public Turing test to tell Computers and Humans Apart is a challenge-response test designed to distinguish humans from bots.

Scrapers can bypass them using third-party CAPTCHA solving services which employ human solvers or, for simpler CAPTCHAs, sometimes with machine learning techniques, although modern CAPTCHAs are very difficult to bypass automatically. Most common programming languages

Should I build my own scraper or use a commercial tool?

This depends on your technical skills, the complexity of the task, and your budget.

Building your own scraper with Python provides maximum flexibility and control but requires coding knowledge.

Commercial tools like Octoparse, ParseHub are often easier for non-coders and offer features like cloud hosting and managed proxies, but come with subscription costs and less customization.

How can I make my web scraper more resilient to website changes?

To make your scraper resilient, use robust CSS selectors or XPath expressions that are less likely to break, implement strong error handling, continuously monitor for breakages, and schedule regular maintenance to adapt to website structure changes or new anti-scraping measures.