Shopping, Big Data, and You: A Look Inside the Changing World of Retail

Here's how the future of retail is changing thanks to technology.

Earlier this month, Lord and Taylor’s iconic flagship store on Fifth Avenue officially ended its reign as a retail palace. Faced with $1B in debt, Hudson’s Bay Company, the department store’s parent organization sold off the property. The new owners? WeWork.

It’s an all-too-familiar tale, one that many devotees of traditional brick-and-mortar retail would tell as a tragedy: unable—or unwilling—to adapt, legacy stores are forced to cut back their operations. The death knell of traditional retail is upon us.

Simultaneously, online retailers keep growing. Ironically, many are strategically pivoting toward traditional brick-and-mortar retail: Amazon continues to open physical locations, the pop-up shop has become a favorite tool of eCommerce darlings like Casper, and Glossier has disrupted the luxe beauty industry with its minimalist, highly curated in-store experiences.

It’s the sort of paradox that flummoxes everyone involved, retailers and pundits alike. Is technology changing how we make purchases, where we make them, or both? If traditional retail is supposedly dying, why do online Amazon sales spike in neighborhoods with physical locations?

It’s an alluring talking point: faced with increased pressure from leaner, more agile online retailers, legacy stores just can’t compete—especially when it comes to attracting millennials and Gen Z customers.

The reality, however, is much more nuanced, and it has a lot more to do with broader economic health (and technology) than it does with generational preferences.

The great retail bifurcation

According to research and consulting firm Deloitte, the past decade has been abysmal for the American consumer. The bottom 40% of earners has struggled to keep up with expenses, while the middle 40% has watched their incomes steadily shrink.

“For 80% of consumers, the last ten years have represented a dramatic worsening of their financial situation,” the firm writes. “For them, it’s been a ‘lost decade.’”

Retail has bifurcated over the past ten years,

This economic slump has had a profound effect on the way most consumers make purchasing decisions. As Deloitte notes, for the first time in ten years, basic necessities account for more than 100% of a low-income family’s budget. At the same time, high-income consumers saw a 4% increase in their discretionary income.

With fewer discretionary funds to spend, low-income consumers prioritize price above all else—which is one reason 58% of them prefer to shop in-store rather than online. It’s only higher income buyers who opt to do the majority (52%) of their buying through digital channels, regardless of their age.

Poorer shoppers are also much more likely to spend their money at discount stores, supermarkets, and—get this—traditional department stores, while better off buyers have more fragmented online shopping habits.

These habits have altered the retail landscape in some remarkable, if under-reported, ways. In an analysis of all public US retailers, Deloitte found that both premier retailers (higher-end stores like Lululemon) and price-focused retailers (such as dollar stores) grew, while balanced retailers’ (like traditional department stores) revenue shrank by 2%. Likewise, price-focused retailers opened an average of 2.5 new stores for every balanced retail store closing.

Not all consumers are taking their retail shopping online.

Deloitte calls this the great retail bifurcation: poorer consumers have less discretionary income to spend, while their better-off counterparts have more. And the split reaches far beyond who is spending money to where they’re spending it. Retail isn’t dying; it’s just no longer growing in the so-called “boring middle” segment, which is where legacy retailers like Sears fit.

“An inconvenient truth to those pushing the ‘retail apocalypse’ narrative, physical store openings actually grew by more than 50% year over year. Much of this is driven by the hypergrowth of dollar stores and the off-price channel,” notes retail expert Steve Dennis, in a Forbes article published in March of last year.

“But there is also significant growth on the part of decidedly more upscale specialty stores and the move of digitally-native brands like Warby Parker and Bonobos into brick and mortar,” he writes.

Selling like it’s 1989

Technology has played a role in which retail outfits are profitable, however. But that role isn’t as simple as, say, enabling brands to have a slick website.

Instead, the real problem lies with the much less sexy, behind-the-scenes technology that powers decision-making.

“From a technology perspective, traditional retailers are really stuck 30-40 years in the past,” says Michael Brand, CEO and co-founder of Dor, a foot traffic analytics company for retailers.

“It’s hard for them to collect the data they need, like actual conversion rates and what people are buying most. Even if they can collect this data, it’s very siloed. It’s challenging to roll data up from each individual location into a central database—and that’s all before they can start crunching the numbers. And if they’re not a team of data scientists, they might not know how to glean meaningful insights from the data.”

These data silos aren’t just an inconvenience; many retailers are functionally making strategic planning decisions about staffing, store openings, and investments in the dark as a result. And that directly shapes profitability.

From a technology perspective, traditional retailers are stuck 30-40 years in the past.

Take something that seems simple, like staffing a store. Too few staff is a recipe for customer dissatisfaction; too many are outright wasteful. Striking the right balance, however, is unbelievably difficult for most retailers, even in 2019; a UNC study found that retailers were under- or over-staffed 86% of the time.

Retailers could easily solve this problem with data about peak shopping times, yet many can’t crack it because they lack the necessary systems and processes for collecting and parsing that data.

This is where technology actually becomes a differentiator. Think about the average direct-to-consumer eCommerce operation, like Quip, Casper, or Allbirds. These retailers have a few distinct advantages—including a narrow range of products and full control over their supply chain—but the most important one is: they have a lot of customer data.

When all of your customer interactions happen online, it’s incredibly easy to collect information about where those customers live, how much they spend on average, what they buy the most, and what marketing strategies are most effective in reaching them. That’s the sort of data legacy retailers have struggled to collect for a long time, which gives up-and-coming sellers like Casper and eCommerce giants like Amazon a competitive edge.

These companies are now harnessing their data to make smarter choices about where to open physical stores. They even use it to decide what to stock and where to place it in the store. If customers frequently buy two products together online, for example, Amazon will place them on the same shelf in a physical store—even if they’re seemingly unrelated, like a vegetable peeler and a compost bin.

Retailers are under- or over-staffed 86% of the time.

This data also makes smaller, direct-to-consumer eCommerce sellers a hot target for acquisition by larger companies, like when Unilever purchased Dollar Shave Club for $1B. When a legacy brand like Serta Simmons acquires a new-kid-on-the-block retailer like Tuft & Needle, they’re often not purchasing the brand just for the product.

“[The merger will give Serta] access to new offerings that will include advanced consumer insights, avenues for generating higher foot traffic, and consumer-centric innovations as key drivers for growth in the evolving market [that Tuft & Needle will develop],” notes the official press release. In other words, what Serta is really buying is Tuft & Needle’s data and tech stack—and the smarter decisions they empower.

Doomed to be boring?

So are mid-market retailers doomed to die a slow death due to a lack of data, a shrinking consumer base, and being squeezed by price-conscious retailers at the bottom and more agile start-ups on the higher end?

Not necessarily.

Back in 2004, as Hurricane Francis barreled towards Florida, Walmart executives sprang into action, led by then-CIO Linda M. Dillman. The state had already suffered through Hurricane Charley a few weeks earlier. This time, the retailer wanted to be ready.

It’s no secret that stores often sell out of certain items before a natural disaster, but Dillman believed there was a smarter way to re-stock the shelves.

“[I knew we could] start predicting what’s going to happen, instead of waiting for it to happen,” she says.

Armed with data from the retailer’s extensive network of in-store devices, her team went to work figuring out what items would be in highest demand. For many, the results were surprising.

“We didn’t know in the past that strawberry Pop-Tarts increase in sales, like seven times their normal sales rate, ahead of a hurricane,” Dillman told The New York Times. “And the pre-hurricane top-selling item was beer.”

These data-driven stocking decisions worked; Walmart sent extra shipments of these in-demand items to Florida stores before the hurricane hit, and most of the products sold exceptionally well.

In order to stay relevant, a retailer must be adaptable, flexible, and open to experimentation.

In this and many other ways, Walmart is a case study in how big box stores can successfully turn the legacy retail model on its head. It’s not quite a dollar store, but it’s not Sak’s Fifth Avenue, either. It’s solidly in the middle, and it’s proving that the middle doesn’t have to be boring.

In fact, more and more legacy retailers—from luxury brands to the world’s largest chain store—have developed better technology stacks that allow them to innovate faster than ever before.

Take makeup emporium Sephora, for example. Though it debuted in 1969 as just another brick-and-mortar retailer, it’s since created an omnichannel customer experience that’s widely hailed as a model for retailers everywhere.

But this tech doesn’t just elevate the retail experience. It also helps the retailer collect more (and better) information.

Sephora’s Beauty Insider program, for example, doesn’t just offer loyal customers rewards; it also serves as a mechanism to bring a customer’s entire purchase history—both in-store and online—into one place. That’s a powerful dataset for any retailer, and it’s gone a long way towards keeping Sephora competitive in an age increasingly dominated by brands like Glossier.  

Innovate or die

As technologies like automation and integration become cheaper and more accessible, retailers across the spectrum will have easier access to crucial data. The question is: will they rise to the occasion and adopt a more tech-forward approach?

“In order to stay relevant, a retailer must be adaptable, flexible, and open to experimentation,” says Brand. “They should be excited to make things better. Revenue is just the result of what’s working.”

Though it might seem like keeping up with technology would be a no-brainer given its effects on profitability, it’s not that straightforward. Many legacy stores continue to close locations and cut costs, rather than invest in tech and innovation.

That’s because flexibility is often rewarded differently for large corporations. One thing most direct-to-consumer eCommerce brands have in common is that they’re new companies. They’re often VC-backed and, as a result, have very different incentives than a publicly-traded legacy retailer with thousands of shareholders. Innovative projects help them attract more funding, rather than being perceived as expensive last-ditch efforts to remain relevant.

Ultimately, it’s not just retail’s technology that’s stuck thirty years in the past; to some extent, their business model and mindset are stuck, too. Until those change, many customers will continue finding new, better, and cheaper ways to shop—elsewhere.

Curious about how retailers are innovating with technology? See how the world’s largest chain store uses AI to offer more competitive wages >