Data is without a doubt the oil to fuel a digital-first country. However, what underpins a country-wide digital shift is the fundamental move from physical, discrete frameworks to connected software and online systems. 

Having static volumes of data sitting in its own silos, without the ability to correlate from multiple sources or the ability to react instantly as it arrives, limits the value that can be derived from the data. 

Data flow between government bodies and businesses

Just as lives are led in the now, so should the technology infrastructure supporting it. With more data being produced in real time due to more systems than ever before, the value half-life of data is reducing exponentially. 

The traditional, silo-based, batch-based forensic approach to data silos is not impervious to this half-life reduction. Stale data, even if stale by a few minutes, results in analytic processes producing stale insights. 

Governments can choose to lead in digital transformation and accelerate the breakdown of these silos to be able to harness fresh data in the moment and recover the stale data before it becomes exponentially less valuable.  

Data-sharing between government statutory boards and corporations can help elevate sectors that are key enablers of a country’s critical infrastructure. 

For instance, in the NSW Government’s future vision for digital identity, citizens can carry their identity and important credentials securely on a mobile device. Individuals can then share the information securely with a third party for service requests or identity verification. 

As government agendas and corporate offerings carefully nurture digital citizenry, this prompts a cycle where citizens progressively enjoy a greater permeation of digital services in their lives.  

A bigger challenge is the ability to connect real time data that is dynamic and always evolving, in addition to state-based data that often does not change. Facilitating interconnectivity of real-time data can nurture a digital nation that is more synchronised in providing more convenient solutions that addresses day-to-day needs. 

For example, the City of Port Philip in Victoria partnered with transport technology scale-up Vivacity Labs and the Bicycle Network, a not-for-profit that seeks to build healthy bike-riding habits, to install traffic sensors that collect bike-rider, pedestrian and vehicle data in real-time to help inform its active travel plans. 

This also enables the council to plan possible physical changes, such as segregated lanes and size of footpaths, to enhance both active travel and improve traffic flow, thereby reducing conflict between transport modes and emissions.  

Securing data while ensuring interconnectivity 

However, with data moved and shared with more parties, it is unsurprising that both citizens and organisations are doubting whether their data is truly private and hence reticent to the shift. Although keeping data in silos may be more secure, there is a need to strike a balance between sharing data and privacy to realise our vision of a digital-first country. 

Strict management of data and granular access control across deployments is necessary to ensure that data is used in the right manner. Access to personal data should only be shared when approved and necessary. This will foster trust and inspire confidence in data infrastructures as the digital-first effort develops further.  

Governments should consider having zero downtime for their data protection services. Threat protection services that are down cannot detect threats, which exposes the rest of the digital-first platforms to security breaches and other forms of intrusion and can lead to devastating consequences for the infrastructure and the reputation of the guardians. 

Yet, the traditional forensic style of data analysis of collection, indexing, and storage all takes time, operational complexity and therefore cost. It means that organisations are discovering attacks or breaches minutes, hours or days too late.  

Streaming data as the foundation for digital-first  

Data streaming – data residing in event-based architectures to respond to a combination of events proactively, reactively and in real time – is an inevitable evolution to keep up with our day-to-day habits. 

Data processing systems need to move from batch processing, which looks at accumulated data at rest, to stream processing, which allows data to be processed as it arrives, leading to real-time recommendations that improve user experiences while the customer is still engaged.  

When faced with cyber attacks, streaming data facilitates the situational analysis of multiple sources such as access time, information downloaded, and general normality for each customer’s behaviour in real time. 

When the likelihood of a fraudulent or malicious act is identified, agencies can actuate a counter response such as shutting access pathways down in fractions of a second. 

Similarly, data that is not synchronised loses real-time relevance for all proactive services. Streaming real-time data enables both government services and businesses to understand their industry, citizens or customers more quickly. 

With more timely insights, they can better serve the community and propel Australia towards a more prosperous and safe society.  

Confluent can help cut across data silos, bringing the old and new data sources together by democratising access to information while maintaining an enterprise-grade level of protection and tenancy of data released from silos, not just the new streams of real-time data. 

I highly encourage you to read our whitepaper on “why data streaming should be part of your data sharing strategy” to explore how we can be a solution to your digital-first conundrum. 

Download our latest whitepaper from Confluent on why data streaming should be part of your data sharing strategy.