Hortonworks Data Platform
Hortonworks Data Flow
EDP implementation & deployment
Event search solution
With many businesses and merchants accepting over 20 different types of payment cards and e-payment options, financial institutions and payment processors are called to process hundreds thousands of transactions per second coming in from around the world, from different channels and in different currencies.
Speed, accuracy and a responsive service is therefore critical for payment processors to remain competitive and offer an integrated solution to their customers.
The customer (Processor) are a leading payments processing services provider, processing millions of transactions from around the world on a daily basis.
As part of their transformation strategy, the Processor wanted to expand their global reach, improve their data analytics and optimisation capabilities, and build solutions to cater to the emerging field of integrated payments. To this end, they started their ambitious project of having an enterprise-wide data lake, aptly named as Enterprise Data Platform (EDP).
With EDP, they wanted to chart a new territory that spans technology and business by having data at the centre of its decision-making strategy. Using data from numerous sources like NAP and ODS across the enterprise, and performing analysis on it, EDP enhances decision-making while providing valuable insights to both internal and external stakeholders. In essence, EDP helps to deliver powerful, embedded predictive analytics to the Processor by providing insight into customer, cardholder, transactional and financial behavioural segments.
True to their strategy of open source software, the Processor chose Hortonworks as their Big Data platform vendor and came to WHISHWORKS, a Hortonworks Gold Consulting Partner and Reseller, to help them make EDP a reality.
We began the engagement running a Proof of Concept, where we started ingesting data from one of the Processor’s existing systems to the Hortonworks Data Platform, and accordingly transform this data for the reporting and analytics tool to consume.
The PoC was implemented successfully and soon after we started bringing in and transforming data from legacy systems across the organisation. This enables the Processor to get a holistic view of how merchants are handling their transaction systems. In parallel, they use EDP internally to ingest log and other operational data and build preventive maintenance solutions within their operational environment.
Hortonworks Data Platform (HDP) along with Hortonworks Data Flow (HDF) provides various technological components to make up EDP services required by the Processor to deploy, integrate and work with massive amounts of data. Each application service requires a different set of ecosystem components to achieve the required functionality.
WHISHWORKS, has architected and built the entire EDP environment, integrating all necessary applications according to the Processor’s requirements, whilst adhering to industry standards and leveraging vendor recommended best practices.
One of the business-critical applications built as part of the EDP was Event Search. The business requirement was to build a search solution to be used by the Processor’s customer support team when handling calls from merchants. The existing legacy solution was very complex to navigate, slow to provide the required information and hard to maintain. WHISHWORKS built the new search solution leveraging Apache Solr, Apache HBase, Apache Spark and Apache Kafka as its core components.
Data is first ingested into Kafka topics through Attunity’s CDC platform – Replicate. Spark based application consumes data from Kafka and writes into HBase after applying business transformations. Finally, data from HBase is indexed into a SolrCloud cluster. A web application has been built on top of the Solr cluster to perform search operations, which get instantaneous responses to customers’ queries.
From a performance perspective, Solr-based search solution is massively faster compared to the legacy Siebel-based application. Besides higher responsiveness, the customer support team can now service a diverse set of queries and requests from merchants, as EDP provides a 360-degree view of each merchant, which wasn’t previously possible due to application silos.
The Event search solution has made the overall customer support operation much more effective, increasing first call resolution rates, decreasing average call time and reducing number of complaints.
Another business-critical benefit achieved through the EDP, is the optimisation of the Processor's internal processes thanks to the visibility it now has across their estate. Gaining access to each system’s performance logs, the Processor can proactively scale, upgrade or replace systems to maintain maximum performance and uptime.
WHISHWORKS consultants have been instrumental in architecting, designing and building EDP’s Hortonworks Data Platform and its integrations with source and other down-stream systems. Our team of specialists has been working alongside the Processor’s team, providing support for the management of the platform and all application systems, whilst enabling them to leverage expert knowledge and industry best practices.
This project has won the Outstanding Big Data Management Solution Award at Computing’s 2018 Big Data Excellence Awards.