Enhancing Intelligent Information Extraction with Minimal Human Intervention

Overview

Semantic search and extractive question answering are the most popular and widely used applications of Large Language Models (LLMs). These applications are critical cornerstones for enhancing the automation for organisations that thrive on data harvesting, curation, aggregation & transformation.  

This article talks about how the power of automated & intelligent information algorithms can be enhanced using a combination of LLMs, semantic search frameworks and a human-in-the-loop using LangGraph, a library built on LangChain. 

Why do we need a human in the loop?

It sounds like an oxymoron when we say that we add a human-in-the-loop to enhance automation. However, the following factors emphasise the positive impact of adding a human to the automation process: 

• Not all information extraction can be automated and there will always be an element of outliers that will not fit into any automation framework. Adding human intervention here will reduce the number of automation cycles trying to achieve perfection. 
• For complex information extraction applications, a small amount of human intervention in terms of approving or rejecting an output can add a valuable feedback mechanism for the language model. 

About LangGraph and LangChain 

LangChain is a framework that directs an agent (human) to perform a certain task and is powered by a language model. The agent runs in a loop to take multiple actions. For example, you can set up an agent to skim through a document on automobiles and extract all the technical specifications of a particular variant. 

LangGraph is built on top of LangChain and can have cycles of multiple loops and multiple actors and improved agent runtimes. For example, we could configure three different agents – 2 subordinate agents collating information from 2 types of sources and one supervisor agent aggregating or filtering the information based on the query. 

Humans-in-the-loop options using LangGraph 

LangGraph proposes two forms of human intervention – Interrupt and Authorise

Interrupt 

• This is the simplest form of human intervention. 
• The user can manually interrupt the output from the language model if required.  
• For example, if the current extraction does not provide any relevant output, the interruption can be triggered, and the application can switch to a different prompt template. 

Authorise 

• This is a more powerful and useful form of human intervention. 
• The system can be configured to hand over control to a human based on certain conditions.  
• For example, if the relevance scores from an extraction are below a particular threshold, a human authorisation of accepting or rejecting the output will trigger a feedback cycle and also improve the output quality. 

In conclusion, by leveraging these human interventions paired with intelligent information extraction, organisations can enhance the accuracy and efficiency of information extraction tasks while maintaining adaptability to handle complex scenarios and outliers effectively. 

References

A recent blog published by LangChain explains the approach of incorporating Humans-in-the-loop with a detailed code walkthrough – Human-in-the-loop with OpenGPTs and LangGraph