Bright lights, smart city, Big Data


Barcelona, the most connected city in the world? 

A smart city is the city of the future. A predicted 60% of the world’s population will be living in cities by 2030. For that to be possible, connected products and services will change the urban landscape forever. From the lift that will move horizontally between skyscrapers whilst collecting data about itself in order to self-diagnose component wear and tear (as well as the elevated preferences of those it transports), to a solar-powered compacting waste-bin that alerts the authorities when it needs emptying.

The development of a smart city relies on Big Data analytics, which offers a real-time view of what is happening in the city at any given time.

For the enablement of a smart city, infrastructure and devices need to be interconnected (a concept known as the Internet of Things, or IoT). The interconnected devices capture and store data in large volumes, and formats as varied as traffic sensor data, smart-meter readings, and Wi-Fi hotspot data, then the smart city needs a method of analysing and making use of all this intelligence.

Smart cities IoT

The right framework for a smart city project

The application of Big Data technology is therefore essential to a smart city project. Big Data platforms such as Hadoop can help collect and capture data from sensors, users, card readers and other sources either via real-time streaming or querying. Big Data tools ensure the effective and insightful utilisation of this data in smart city applications, and Big Data management, including the development and execution of architectures, policies and procedures, can be used to manage the full application data life cycle. Analysing smart city data in near real-time requires a scalable and reliable IT platform that combines high performance computing capability for executing the different data-intensive applications with highly fault tolerant stream processing.

Older IT platforms are not scalable nor robust enough to handle the volume, velocity and variety of this data. The smart city will rely on Big Data platforms like Hadoop and the advanced data management features of this technology which recognise different data formats, their sources, structures and classifications.

Compare the capture of data formats in earlier IT platforms like relational database management systems (RDBMS), which were designed to store structured data. The RDBMS model is pre-designed with columns and rows of expected data (schema), no other form of data can be processed by an RDBMS. Unlike the Hadoop platform, where data is ingested in free-form (as created by users) i.e. clickstream, weblogs, twitter feeds, blogs, videos etc.

This is where Big Data features like HDFS, Hive and HBase shine. As data is tipped into the Hadoop Distributed File System (HDFS), schemas are created as the data is queried. HBase and Hive are NoSQL databases built on Hadoop, which provide access and consistency for large amounts of unstructured and semi-structured data - potentially billions of rows, multiplied by millions of columns of schema.

Smart city projects have taken advantage of Hadoop infrastructures to enable rapid analysis of data in much larger volumes when compared to RDBMS technology. The opportunity provided by Big Data analytics in smart cities enables faster decision making, increased productivity, reduced energy consumption, more efficient waste management, and better routing of traffic to reduce traffic congestion.

Smart city projects running Big Data analytics

The City of Barcelona in Spain wanted enhanced services for its people and businesses. To achieve this, they used Windows Azure, HDInsight, and SQL Server to collect, analyse, and generate a near real-time business intelligence dashboard for Big Data collected from social media feeds, GPS signals, parking systems, public transport, street lighting and waste management. The city realised significant cost savings, improved the quality of life for residents, and made the city a home for the Internet of Things.

In Songdo South Korea, the garbage collection process generates data from residents using a chip card in the garbage containers as part of a waste management monitoring program. Each house has a garbage disposal unit and all the city’s garbage is sucked into underground pipes which takes it to an automated garbage collection plant. All apartment buildings and offices in the city are connected to the system. The data is processed in a Hadoop data lake using Flume, a framework for populating Hadoop with data via agents on web servers, application servers, and mobile devices.

To learn more about the connected world read API’s in the IoT.

If you would like to find out more about how Big Data could help you make the most out of your current infrastructure while enabling you to open your digital horizons, do give us a call at +44 (0)203 475 7980 or email us at

Other useful links:

 5 Benefits of Deploying Hadoop

The demand for Big Data experts: Is there a real shortage?

What does it mean to be a data driven organisation?

Survey Report: The State of Big Data in the UK 2017/2018

Recent Posts