4 Steps to Optimise Data Engineering in FMCG and Retail Sectors

A US-based consumer goods company, which makes low-calorie, high-protein ice creams, wanted to make a grand entry in the UK market. Its objective was to make it available across Tesco outlets, create awareness about the brand, encourage consumers towards a healthy eating trend, and grow its business and revenues.  

The company turned to a well-known external media agency to implement its objectives, the core idea being, the campaigns would run mostly based on data-driven insights about its potential consumers.  

The result? Within six months, it became the second most popular luxury ice cream brand. It engaged with 66% of the target audience who were former consumers, or are new to the category. Eventually, it became the UK’s third largest ice cream brand within a year, and contributed to 1% of the luxury ice cream and sorbet industry. 

Making data-backed business decisions is anything but new in the global retail and FMCG industry. But, one could argue that this became a norm only post pandemic. Last year, Nielsen released a report which showed that pre-pandemic, only 17% of the European retailers they surveyed were tech visionaries. 31% of them felt tech is necessary only on a need-to basis.  

In fact, businesses that adopted tech into its decision-making process had siloed data pockets across the organisation, with data experts spending more time in structuring data, and less time in deriving holistic insights from them.  

Today, post pandemic, the adoption has increased significantly, with retailers and FMCGs inculcating data-backed decision-making at every level of business, from operations to marketing. 

But, adopting big data into business is hardly an overnight process. It requires the business to go through several steps including identifying key business objectives, centralising data assets from across channels and departments, implementing AI and machine learning technologies to collate, churn and structure the data, and a skilled team of data scientists and data experts across teams to generate insights from holistic, real-time data. 

Let’s look at critical steps retail & FMCG businesses need to take to build an effective data engineering pipeline. 

We recently published a blog about why analytics-driven decision-making is key for ecommerce businesses. You can read more about it here. 

Identify Sources of Data 

In the retail and FMCG sector, there are typically four levels of data that are relevant; 

Operational data 

This includes PoS transactions, purchase orders, inventory, sales across each store and sales generated online, number of customer footfalls and purchases across stores and online sites, excess stock info, and more. 

Competitor data 

Includes sales and market growth of competitor stores and digital sites, their market share, their campaigns and promotional activities, pricing strategies, performance across stores and online outlets, and more. 

Marketing data 

Includes customer engagement with the campaigns and promotions run by the brand, engagement to conversion data, feedback, reviews and more. 

Weather data 

Includes economic and political conditions, weather reports and such.

Converting Data into Consumable Formats 

Businesses collect both structured and unstructured data from various sources such as legacy systems, modern data systems like web servers and APIs, spreadsheets, social media channels and more.  

These require data experts to run them through quality checks, and structure them across stages, to eventually make them consumable by data scientists and teams across the organisation.  

Some best practices to follow during data formatting and structuring are; 

• To ensure that the file name and categories are clear and distinct 
• To create metadata for each data set 
• To identify the best way to store your data. This can depend on the size of your organisation. 
• To document each set of data you have, so that you can refer back to why that data was stored and how it will be relevant to your business 
• To put together a data practice in place so that data management best practices trickle down from the leadership level to the last employee in your organisation 

• To ensure proper security and privacy measures are in place 
• To invest in a compatible and efficient data management software 

Monitor Data Governance 

Enough is being said about the need for data protection, and the consequences businesses would face in the event of a data breach. While studies reveal that consumers are open to getting personalised recommendations based on their preferences and behaviours, businesses still need to ensure that they seek the consent before using the data, and ensure that their processes are falling within Government-set guidelines.  

Aim for Digital Skilling from Get Go 

A Merit expert says, “Even though a business may be ready to implement data into its decision-making process, understanding and applying that data requires certain skills, which cannot always be outsourced to a third party. Businesses need to ensure that its employees are trained in data management and learning how to use technology.”  

For example, Whole Foods equipped its store employees with the skills to monitor and identify how other Whole Foods stores are performing in the city or town, and extracting highly relevant data points like the time period when footfalls are highest and lowest, products that are in greatest demand, the closest warehouse to secure inventory and such. 

In conclusion, as more and more retailers face the pressure to grow their business and consumer base amidst rising inflation, economic conditions, and consumer demands, experimenting with data and identifying new ways to apply it in day-to-day operations and planning can go a long way in helping them remain competitive, profitable and consumer-oriented. 

Merit’s Expertise in Retail Data and Intelligence   

Our state-of-the-art eCommerce data harvesting engine collects raw data and provides actionable insights   

• Three to four times faster than standard scrapers   
• At lower cost   
• With Increased accuracy (up to 30% compared to standard scrapers)   

Our powerful, new scraper engine can gather massive data sets from multiple sites and geographies in real-time so you can stay informed on customer behaviours and market trends.   

Merit’s eCommerce data engine provides a high degree of confidence in insights generated from analytics – thanks to confidence in the data quality and access to enriched data.   

To know more, visit: https://www.meritdata-tech.com/service/data/retail-data/