Here’s What Ethical Web Scraping in Healthcare Looks Like

Web scraping, as the name suggests, is the process of scanning reams of publicly available information on the Internet, to derive insights into who is looking for what, and using them to drive better business decisions.  

This process is usually performed by bots or other tools, and it simulates how a human surfs the web. For example, marketers extensively use web scraping to find out what their customers or potential customers are searching for or talking about on social media platforms, and placing ads or sending emails to gently nudge them towards a conversion. 

Of course, with emerging increasing awareness around responsible use of online data, thanks to the introduction of data privacy laws like GDPR, web scraping has become more ethical, keeping in mind the preference of the individual and how personalised they want their interactions to be. 

The types of data scraped for healthcare intelligence 

In healthcare, the insistence on it being ethical and transparent is all the more important because of the nature of data being scraped. Usually, healthcare web scraping includes collecting data on; 

• Doctors and healthcare professionals and their specialities by location, popularity, price, facilities and such 
• Reviews of hospitals, insurance companies, pharmacies etc. 
• Regulatory and governance policies, medical insurance policies and claim conditions in place across different players in the ecosystem 
• Diseases and prescribed medications, cost of treatment, treatment and cost comparisons and such (usually available as public health records) 

• Medical devices and its adoption, penetration and effectiveness, and more 

Use cases for healthcare data  

Organisations in the healthcare ecosystem can use these data points on multiple fronts; they can aid in clinical research, enable better decision making for patients (when choosing a healthcare provider, a treatment plan etc.), it can help recognise trends and prepare for epidemics, it can bring efficiency into billing, treatment, insurance claims, and general healthcare practices and processes, it can help organisations analyse their competition, be aware and prepared with changing economic conditions, and more. 

Case study: The quality of Care for HIV patients in North Carolina 

For example, the National Library of Medicine performed a study, where it analysed the quality of care provided for incarcerated HIV patients in North Carolina (U.S). It began this study on the premise that, though there is enough research to prove that HIV medications improve patient health and prevent further transmission of the disease, 40% of the HIV population still doesn’t have access to proper medical care to achieve this outcome wholly. And, one in six of these HIV patients spent a portion of their life in incarceration.  

Hence, the goal of the study became to identify HIV patients who have been incarcerated, and the care that was being provided for them before, during and after incarceration.  

Because the jails in North Carolina operated independently, only 29 of the 97 jails in the State had public information on their websites on inmates, their name, age, place of birth, crime and jail term.  

The team combined data from the public jail records of these 29 jails (through web scraping), and combined it with the data available in the State Division of Public Health on the name, age and HIV care provided for patients the State, to identify how many of them were jail inmates and what care they were being provided pre, during and post jail term.  

In the context of their study, they kept the scraping ethical by removing the individual names and personal identities of inmates, and simply using these as general (or average) population data. 

In another example, Meta uses web scraping to analyse posts and send an intimation to first-responders if they identify with keywords like suicide. 

Ethical Best Practices in Healthcare Web Scraping 

There aren’t any specific laws related to healthcare web scraping. These practices apply to the process across industries and objectives.  

A Merit expert adds, “For instance, in the UK and European region, the Digital Services Act legalises the use of publicly available content, but it becomes illegal under GDPR when it is used against the consent of the individual, or is used to harm the individual or organisation from where the data has been collected.” 

With this in mind, let’s look at some points businesses need to consider when performing web scraping; 

• They should ensure that the data from scraping is not harming the business or website performance. On the business side, websites can ensure that they develop API with information that web scrapers can ask and collect. 

• They should only scrape data that they need, and they should limit the number of requests per second, so that the owner doesn’t perceive it as a DDoS attack 
• They should read the data permissions policy of the website they are scraping 
• They should not collect information that can link to a specific person/ individual  
• They should have a technical team on board to help them read the fine print of data collection policies and laws 
• They should identify themselves (usually with a user agent string) when scraping a website so that the website owner can get in touch with them if needed 

Merit Data & Technology: A Trusted Web Scraping & Data Mining Partner, With a Deeply Ethical Approach 

At Merit Data & Technology, our team of data scientists have extensive, in-depth experience in working with data to facilitate web scraping in an efficient and effective manner, leaving you to focus on the core areas of your business and improve your team’s productivity and efficiency using the data and analytics we provide. Our data scientists understand your data needs and create customised tools to deliver the right data in the format you need. They scale up and scale down the data collection process based on your business needs, and validate data quality before it is used for analytics and decision-making. 

To know more about our web scraping technologies and practices, visit https://www.meritdata-tech.com/data/