Why Text Analytics is Critical for Industry Data and Market Intelligence

In this blog, we highlight the role of garnering insights from text data for better market research and industry-level intelligence. 

Even before the term ‘Analytics’ became a part of business jargon, most business leaders did use data for decision-making. Those days, it was called ‘number crunching’ because data was mostly numerical – sales data, financial numbers, budget information, etc.  

With the advent of Business Intelligence (BI) tools, analysis became easier but was still restricted to numerical data. While this did enable getting a sense of the market trends, there was no way for businesses to capture “qualitative intelligence”.  

Therefore, in addition to number crunching, decision-makers used their instinct and gut while analysing qualitative information.  

But now, there is infinitely more information a marketer has at his or her disposal. Of course, there are hard, factual numbers from budgets and spends, the number of customers acquired, the impact of marketing and ad campaigns on sales, etc. In addition to that, the use of BI tools enables marketing teams to distill insights on what worked, what didn’t and even predict performance or find new areas or ways to build campaigns.  

Add to that the value of comments posted on social media, feedback shared on customer service calls, and reviews posted on e-commerce sites? Today, marketers can feed this information into their analytical engines. In fact, experts believe that unstructured text data constitutes nearly 80% of the data with structured data being 20%. And, that is where text analytics comes into play.  

Text Analytics for Market Expansion  

Text analytics solutions are fast gaining importance and the market for such solutions globally is expected to grow from USD 5.46 billion in 2020 to USD 14.84 billion by 2026, at a CAGR of 17.35%. Text analytics is being widely used for predictive analytics, market research, fraud management, cybercrime prevention, and risk management, among others. 

The Role of Text Analytics at Industry Intelligence Firms  

For industry intelligence firms, text analytics has become all the more crucial. These firms are looking for industry-level insights to spot macroeconomic trends, geographical or political risks, and technological advancements to name a few areas. For industry intelligence firms, the customers are various stakeholders within the industry who’re relying on them to spot industry-level threats and opportunities.  

Let us take the automotive industry as an example. Automotive intelligence firms are unearthing insights about the future of mobility, the rapid increase in the potential of electric vehicles, emerging technologies crucial for the connected car of the future and how underlying technology is being developed to build semi-autonomous and fully-autonomous vehicles.  

Some of these shifts may happen sooner rather than later – like the rapid shift we’re witnessing in the EV market, while some others (like autonomous, self-driving vehicles) may take time. Having said that, it is already proven that autonomous vehicles are working well in closed environments, like golf carts used in a golf course, for instance.  

All these insights are not coming from numerical data, but by culling out information buried inside text documents or videos or even photos.  

Real-life Uses of Text Analytics for Intelligence in Europe 

At Merit, we work closely with several market and industry intelligence firms in Europe and it has become critical for all of them to cull insights from chatter on social media, comments posted on review sites, and even statements made by top industry leaders buried inside articles or annual reports of companies.  

Sources for text analytics tools  

Text analytics tools enable businesses to gain actionable insights from unstructured text sources including: 

• Product or service reviews posted on websites 

• Tweets and other social media chatter  
• Client interactions or interviews 
• Earnings calls or analyst conversations  
• Quotes provided during interviews 
• Blogs and whitepapers  

• Annual reports  

The advancements in AI and machine learning are further spurring the growth of text analytics – making it easier to use NLP (natural language processing) technology to capture insights from text inputs. Apart from text documents, data is also being extracted from audio and video, stock ticker data, and financial transactions. 

Using text analytics solutions, businesses can automate data extraction from large volumes of unstructured text. When combined with data visualisation tools, this can help to set the context behind the numbers, conduct sentiment analysis, and make better decisions.  

A data science expert at Merit, adds this: “An advanced text analytics tool needs to go beyond semantics to set the context and understand the true meaning behind the review or comment. Most languages are complex and cultural influences also may convey a different meaning from what was intended. Sometimes the frame of mind of the writer may influence the choice of words – where the same word can mean something different when used sarcastically, especially in reviews or social media comments. Therefore, for businesses to truly make sense of the textual data, context-setting also becomes very important.”  

The benefits of using text analytics to acquire business intelligence insights 

With text analytics, businesses can: 

• Understand customer expectations better, respond in a timely manner and improve delight 

• Identify issues with their products and services and improve quality, functionality, etc. 
• Improve the outcome of market research 
• Increase brand reputation by reacting swiftly to what is being said about them in the marketplace 
• Device dynamic pricing strategies based on market trends 

For market intelligence firms that cover industry-wide insights, threats and opportunities, it is a no brainer to embrace text analytics, and make it a core part of the BI stack.  

A Solution for forecasting consumer and design trends using text analytics 

One of our clients, a world-leading authority on forecasting consumer and design trends, required information from millions of fashion products spanning multiple categories and sub-categories to be collected, aggregated, and reported within 24 hours of them being published online. 

Merit explored the full scope of the challenge and helped implement an AI-driven solution with two primary goals: one, decrease cost of data gathering, and two, speed by the process. Our data engineering team deployed cutting-edge data collection robots integrated with machine learning tools to gather and refine around 10 million records per day. Our team of expert researchers further augmented and enriched the data gathered using automation, but also with some primary and secondary research.  

For this client, we collected and managed data in real-time, with high velocity, transforming public domain data into extremely useful information and intelligence. In all our industry intelligence engagements, we’re completely GDPR compliant, transparent and audit-ready. 

Merit Group’s Expertise in Text Analytics and Data Harvesting 

At Merit, we are specialists in collecting high-volume, accurate industry-specific data, at scale and speed.  

Driven by our own data engine Data Xtractor, Merit enables rapid data collection, transformation and ingestion from a diverse range of disparate online sources. Our solutions help some of the world’s largest intelligence brands seamlessly deliver data and insights to their end customers, including:  

• Collecting attributes of retail products on eCommerce platforms  
• Audience data for events or marketing activities  

• Delivering curated content from thousands of online documents or PDFs  
• Aggregate millions of specialised, industry-specific data points 

Our data engineers make it easy for industry intelligence firms to automate the following processes:  

• Harvest, aggregate and manage data, images, and narrative content from thousands of web sources  
• Handle data in various formats – HTML, APIs, JSON, Excel, PDFs, PowerPoint  
• Refine the raw data into a consistent and searchable format  
• Enhance data with additional research to deliver value-added insights  
• Setup the underlying data and information flow using the right data lake and date warehousing architecture  

To know more about text analytics, data harvesting and aggregation services, visit: https://www.meritdata-tech.com/service/code/data-harvesting-aggregation/