4 Major Challenges in Healthcare Data Harvesting

In an earlier blog, we had looked at why data harvesting is pertinent for healthcare companies. To put it briefly, it improves clinical decision making, it enables healthcare providers to make quicker diagnosis, it improves administrative processes, thus fostering better customer relationships, it makes healthcare more accessible, and it enables the board to forecast the future and make more informed decisions.  

Given the numerous benefits of using data in healthcare, why are companies still struggling to implement it? The simple answer is, in an industry like healthcare, it’s not about collating and drawing inferences from data from within the organisation alone. It requires an entire ecosystem to come together, collaborate and exchange data and information to drive truly actionable insights. Let’s understand this with an example. 

Individual health records with data across multiple organisations 

Let’s say there’s a patient named Jane Doe. She visits a pharmacy in her city and buys over the counter medications for a common flu. A year later, she’s on a business trip to another city, falls ill, gets hospitalised and takes treatment there. A few more years later, she’s on a vacation and she visits a doctor about an allergy. At some point, if she visits a hospital to treat a major condition – like, say diabetes – the hospital will only have as much information as they get that one time.  

At best, if she has visited the same hospital more than once, they will have patient history based on her past visits to this particular hospital. Whereas, if the healthcare ecosystem in her country had implemented a holistic data harvesting mechanism, every pharmacy visit, every medical claim, and every major or minor illnesses she has had up to that point would be on a single patient record, which would help healthcare providers make more informed decisions about the ideal treatment plan for Jane Doe. 

While this is a straightforward example, it becomes even more complex when governed by strict data regulations, and layered with implementation of technologies that require the healthcare providers and administrative staff to be trained adequately to operate.  

We’ll look at the challenges faced in healthcare data management in more detail. 

Healthcare Data Silos Across Organisations 

One of the first challenges the healthcare ecosystem faces is data silos. Like we said earlier, it’s not just data silos within an organisation, but it exists within an ecosystem itself, which is a bigger problem to tackle.  

If you look at a traditional health information system (HIS), while it collects, stores and manages patient and administrative data, it is usually designed to store information department-wise.  

Moreover, the data is available in multiple formats like lab reports, clinical notes, images, videos and the like. Unless there is an NLP or AI technology that is collating and presenting this data in a structured manner, it’s difficult for healthcare institutions to make proper sense of pockets of data.  

The National Library of Medicine had published an interesting study on how eliminating data silos can help mitigate missing person incidents among dementia patients. The study revealed that almost 60% of dementia patients go missing at least once during the course of their disease. And, if there were a national strategy for the collection of information related to people with dementia, it would cut down the time and resources allocated to search and rescue them, and prevent similar future mishaps from taking place. 

Unstructure data in varied formats 

The second challenge healthcare faces is data being available in a wide range of formats. In an earlier blog, we had looked at the two types of data; structured and unstructured. The former typically includes elementary information like patient name, age, height etc., and the latter includes images, clinical notes, scan reports, lab reports and the like.  

In the absence of NLP technologies, healthcare providers will find it challenging to bring both types of data together into a structured, readable format that they can derive insights from. 

Untapped health data from wearable devices 

The third challenge is the evolution of health apps and wearable devices. In 2022, the wearable medical devices market was valued at USD 78 billion, and with the rise of chronic illnesses like diabetes and cardiac diseases, the industry is set to grow at a CAGR of more than 24% between 2023 and 2032.  

While these trackers are personalised and promote healthy living in people, this data is critical when to studies, clinical trials or even evaluating an ideal treatment plan. And, the healthcare ecosystem is finding it challenging to pool this data because the data is highly governed and one small compliance misstep can cost the organisation heavily. 

Healthcare data governance, data protection and regulations 

The last challenge is data harvesting within the Government framework and regulations. In the UK, the healthcare laws are governed by the Data Protection Act (2018), the GDPR, and the Common Law Duty of Confidentiality (CLDC).  

On a broader perspective, these laws require any use of patient data or any healthcare data to be lawful, fair and transparent. At the organisation level, the laws require individual organisations to implement strict data regulation policies and guidelines applicable to every staff member, and in the event of data sharing with another organisation, the laws require that the organisation draw up a data sharing agreement to determine who will be responsible for the privacy and security of this data.  

A Merit expert says, “While these are just a handful of examples, because healthcare is a highly regulated industry, and loss, theft or misuse of data can have a  large impact on organisations, creating a data ecosystem and sharing data across organisations requires detailed data governance policies which can be complex and time consuming. At times, organisations may refuse to share data for the simple fear of not wanting to get into regulatory constraints through a lack of understanding.” 

Despite these challenges, healthcare organisations can make a good start by simply collecting existing data and building NLP-based algorithms to derive meaningful insights. Creating a universal healthcare data system can be a process that takes place gradually, as technologies evolve and regulatory bodies develop more clear, precise laws around data usage and sharing within the industry. 

Merit’s Expertise in Healthcare Data Harvesting 

Our state-of-the-art data harvesting engine collects high-volume, industry-specific data at 4 times the speed, with 30% more accuracy than normal scrapers, at a lower cost and with the quality control from seasoned data experts. 

Our solutions help some of the world’s largest healthcare brands seamlessly deliver data and insights to their end customers, including: 

• Delivering curated content from thousands of online documents or PDFs 
• Aggregating millions of specialised, industry-specific data points 

To know more, visit: https://www.meritdata-tech.com/service/code/data-harvesting-aggregation/