Inside eBay’s 90PB data warehouse

 

Engineers build analytics-as-a-service.

eBay has spent the past two years transforming its data analysis and reporting capabilities so front-line staff may help themselves to its massive data store.

The e-commerce giant stores almost 90PB of data about customer transactions and behaviours to support some $3500 of product sales a second.

Data is stored in three systems, with about 7.5PB in a Teradata enterprise data warehouse, 40PB on commodity Hadoop clusters and 40PB on ‘Singularity’: a custom system for performing deep-dive analysis on semi-structured and relational data.

eBay analytics platform and delivery director Alex Liang said the site’s growing popularity and usage trends had put a strain on his Shanghai-based engineering team.

“We noticed that smartphone and tablet users visit eBay more times every day than PC users,” he told a Teradata customer conference in Sydney on Thursday.

“We get more visits, which means we get more data, which means we have to process more data. That’s a problem we’ve been resolving for the past two years.”

As of last week, eBay had 500 million live auction listings, split into more than 50,000 categories.

The site has more than 100 million active users, generating up to 100TB of new data each day to be stored and used by more than 6000 eBay staff.

Users range from expert data scientists to non-technical business people and about 50 executives who need access to top-line reports.

“We need to make sure that the people who run the business, people who are doing the innovation, have direct access to the data instead of having people in the middle,” he said.

Liang likened eBay’s previous data analysis ecosystem to an “Ikea job interview” in which applicants would need to build their own chairs.

“When you first come to eBay, you don’t even know where to get data because there is data everywhere,” he said.

“We say, ‘we have all the data there, we have all the tools there’. So when you come to the company we say ‘just use the data’. But the problem is much more complex.”

Building a consistent, user-friendly ecosystem

eBay’s data analysis and reporting capabilities have come a long way since the late 1990s, when it had no data warehouse and used Microsoft Access for financial reporting, Liang said.

Staff now use a custom ‘DataHub’ platform to access a wide range of data access tools, including Microsoft Excel, visualisation tool Tableau, Oracle and MicroStrategy software, SQL and purpose-built applications.

Liang said DataHub represented “three years of heavy investment” during which it had adopted more user-friendly features like a search bar and categories.

A second platform, QuickStrike, offers a range of reporting dashboards to ensure that all users see a consistent set of metrics and key performance indicators.

Before QuickStrike was introduced, Liang said various business areas were using seven versions of what should have been the same metric across the globe.

“[It was] very tough for people to find the right metrics and reports,” he said. “People just go and do their own metrics and reports based on their assumptions and understanding of the data.

“The challenge is that in many cases, they don’t know which metrics and reports are old, which ones are new, which ones are accurate.

“So in many cases people would sit around a table discussing a business problem and think they’re talking about the same thing but … their reports had their own definitions.”

Ebay made two teams responsible for QuickStrike data quality and governance: a technology group and a dedicated team of business analysts.

The QuickStrike dashboard template took a year to design, Liang said, noting that eBay staff were sharing reports via email as recently as last year.

eBay staff also have access to a third tool, Metrics Explorer, to dig further into any business problems and discover potential solutions through deep data analysis.

Liang said eBay was committed to a multi-platform, self-serve data analytics environment. He said it would look to add a 1PB in-memory database to its trio of platforms in the near term.

“The future will be live,” he said. “You can’t drive your car by looking at the back … when you run a business, you can’t just look at what happened yesterday. You must be able to predict the future.”

Copyright © iTnews.com.au . All rights reserved.


Inside eBay’s 90PB data warehouse
 
 
 
Top Stories
Frugality as a service: the Amazon story
Behind the scenes, Amazon Web Services is one lean machine.
 
Negotiating with the cloud email megavendors
[Blog post] Lessons from Woolworths’ mammoth migration.
 
Qld govt to move up to 149k staff onto Office 365
Australia's largest deployment, outside of the universities.
 
 
Sign up to receive iTnews email bulletins
   FOLLOW US...

Latest VideosSee all videos »

The great data centre opportunity on Australia's doorstep
The great data centre opportunity on Australia's doorstep
Scott Noteboom, CEO of LitBit speaking at The Australian Data Centre Strategy Summit 2014 in the Gold Coast, Queensland, Australia. http://bit.ly/1qpxVfV Scott Noteboom is a data centre engineer who led builds for Apple and Yahoo in the earliest days of the cloud, and who now eyes Asia as the next big opportunity. Read more: http://www.itnews.com.au/News/372482,how-do-we-serve-three-billion-new-internet-users.aspx#ixzz2yNLmMG5C
Interview: Karl Maftoum, CIO, ACMA
Interview: Karl Maftoum, CIO, ACMA
To COTS or not to COTS? iTnews asks Karl Maftoum, CIO of the ACMA, at the CIO Strategy Summit.
Susan Sly: What is the Role of the CIO?
Susan Sly: What is the Role of the CIO?
AEMO chief information officer Susan Sly calls for more collaboration among Australia's technology leaders at the CIO Strategy Summit.
Meet the 2014 Finance CIO of the Year
Meet the 2014 Finance CIO of the Year
Credit Union Australia's David Gee awarded Finance CIO of the Year at the iTnews Benchmark Awards.
Meet the 2014 Retail CIO of the Year
Meet the 2014 Retail CIO of the Year
Damon Rees named Retail CIO of the Year at the iTnews Benchmark Awards for his work at Woolworths.
Robyn Elliott named the 2014 Utilities CIO of the Year
Robyn Elliott named the 2014 Utilities CIO of the Year
Acting Foxtel CIO David Marks accepts an iTnews Benchmark Award on behalf of Robyn Elliott.
Meet the 2014 Industrial CIO of the Year
Meet the 2014 Industrial CIO of the Year
Sanjay Mehta named Industrial CIO of the Year at the iTnews Benchmark Awards for his work at ConocoPhillips.
Meet the 2014 Healthcare CIO of the Year
Meet the 2014 Healthcare CIO of the Year
Greg Wells named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at NSW Health.
Meet the 2014 Education CIO of the Year
Meet the 2014 Education CIO of the Year
William Confalonieri named Healthcare CIO of the Year at the iTnews Benchmark Awards for his work at Deakin University.
Meet the 2014 Government CIO of the Year
Meet the 2014 Government CIO of the Year
David Johnson named Government CIO of the Year at the iTnews Benchmark Awards for his work at the Queensland Police Service.
Q and A: Coalition Broadband Policy
Q and A: Coalition Broadband Policy
Malcolm Turnbull and Tony Abbott discuss the Coalition's broadband policy with the press.
AFP scalps hacker 'leader' inside Australia's IT ranks.
AFP scalps hacker 'leader' inside Australia's IT ranks.
The Australian Federal Police have arrested a Sydney-based IT security professional for hacking a government website.
NBN Petition Delivered To Turnbull's Office
NBN Petition Delivered To Turnbull's Office
UTS CIO: IT teams of the future
UTS CIO: IT teams of the future
UTS CIO Chrissy Burns talks data.
New UTS Building: the IT within
New UTS Building: the IT within
The IT behind tomorrow's universities.
iTnews' NBN Panel
iTnews' NBN Panel
Is your enterprise NBN-ready?
Introducing iTnews Labs
Introducing iTnews Labs
See a timelapse of the iTnews labs being unboxed, set up and switched on! iTnews will produce independent testing of the latest enterprise software to hit the market after installing a purpose-built test lab in Sydney. Watch the installation of two DL380p servers, two HP StoreVirtual 4330 storage arrays and two HP ProCurve 2920 switches.
The True Cost of BYOD
The True Cost of BYOD
iTnews' Brett Winterford gives attendees of the first 'Touch Tomorrow' event in Brisbane a brief look at his research into enterprise mobility. What are the use cases and how can they be quantified? What price should you expect to pay for securing mobile access to corporate applications? What's coming around the corner?
Ghost clouds
Ghost clouds
ACMA chair Chris Chapman says there is uncertainty over whether certain classes of cloud service providers are caught by regulations.
Was the Snowden leak inevitable?
Was the Snowden leak inevitable?
Privacy experts David Vaile (UNSW Cyberspace Law and Policy Centre) and Craig Scroggie (CEO, NextDC) claim they were not surprised by the Snowden leaks about the NSA's PRISM program.
Latest Comments
Polls
Which bank is most likely to suffer an RBS-style meltdown?





   |   View results
ANZ
  21%
 
Bankwest
  9%
 
CommBank
  11%
 
National Australia Bank
  17%
 
Suncorp
  24%
 
Westpac
  19%
TOTAL VOTES: 1435

Vote