I have closely been researching the case for data sharing in Government for several years now as the topic intrigues me because the benefits are self-evident.
However, the ability to enable data sharing at all levels of Australian Government, with the backdrop of constantly changing systems and departments, privacy legislation and cybersecurity risks is at best very hard going, at worst, being done via emailing 100,000’s of copies of data embedded in Excel spreadsheets around.
This is not scalable, best practice and nor is it sustainable to produce the sovereign benefits for the Australian citizen. Anyone in Government will tell you that data sharing can play a vital role in efficient governance and service delivery by enabling informed decision-making, enhancing service delivery, providing a holistic understanding of citizens and optimising resource allocation. It can be a powerful tool that supports evidence-based policies, improves public services, and ultimately contributes to the betterment of society as a whole.
So, why is it so hard for Australian departments and agencies to share data? Through many conversations with key stakeholders in Government over the years and many (MANY) hours of research, below are some of the key obstacles (some are obvious) I have uncovered on why data sharing is so challenging in Australian Government.
Siloed Data and Lack of Interoperability: Government departments often operate in silos, making it difficult to share and integrate data seamlessly.
Legal and Regulatory Barriers: Data sharing encounters legal and regulatory challenges, such as privacy laws and restrictions on sharing sensitive information.
Data Quality and Consistency: Ensuring data quality and consistency is a challenge when different departments have varying data standards and collection methodologies.
Privacy and Security Concerns: Sharing data raises privacy and security concerns, requiring appropriate safeguards and compliance with regulations.
Technological Infrastructure and Compatibility: Incompatible infrastructure and legacy systems can hinder data sharing, necessitating upgrades or integration efforts.
Governance and Trust: Establishing clear governance structures and fostering trust among stakeholders is crucial for successful data-sharing initiatives.
There are three interdependent overarching complexities at play:
- Government Data Policy
- Data Architectures
- Change Management that causes downstream disruption
How can Government Leaders not be overwhelmed and address the sustainability of data sharing at scale despite these complexities?
Perhaps an approach that Australian Governments can look at is focusing more on connecting datasets via data virtualisation which has been successfully deployed globally by the US NATIONAL NUCLEAR SECURITY ADMINISTRATION (in partnership with Pantex) when sharing sensitive information on nuclear weapon product lifecycle.
Data virtualisation allows government departments and agencies to access and query data from multiple sources in real-time, without physically moving or copying/replicating the data (not needing to copy data alone will address the many challenges I have listed above). Here are some key points on how data virtualisation can help Government departments and agencies share data:
Integration of Disparate Data Sources: The Australian public sector typically deals with a vast amount of data residing in diverse systems, such as government agencies, departments, and external partners. Data virtualization allows for the integration of data from these disparate sources, regardless of their location or format. It provides a unified view of the data, making it easily accessible and eliminating the need for time-consuming data integration projects.
Real-Time Data Access: Timely access to accurate data is crucial in the public sector for decision-making, policy formulation, and service delivery. Data virtualisation enables real-time or near real-time integration of data from various sources. This ensures that public sector employees have access to the most up-to-date information when they need it, facilitating faster and more informed decision-making processes.
Enhanced Data Governance and Security: Data governance and security are of utmost importance in the public sector, given the sensitivity of the information involved. Data virtualisation platforms, such as Denodo, provide robust security features to protect data privacy and comply with regulations. It offers granular access controls, data masking, and encryption capabilities to safeguard sensitive information. Additionally, data virtualisation enables the implementation of data governance policies, ensuring data quality, lineage, and auditing, which are essential for maintaining data integrity and accountability.
Agile Data Delivery: Data virtualisation allows public sector organisations to quickly respond to data requests and deliver insights to stakeholders. It eliminates the need for time-consuming data replication or ETL processes, reducing the time and effort required to access and analyse data.
Self-Service Data Discovery and Analysis: Data virtualisation platforms often provide self-service capabilities, enabling business users in the public sector to explore and discover data on their own.
Cost Efficiency: Data virtualisation can lead to cost savings in the public sector. By virtualising data instead of physically moving or replicating it, organisations can optimise their IT infrastructure and reduce storage requirements.
By leveraging data virtualisation, the Australian public sector can overcome common data sharing challenges such data integration challenges, ensure data governance and security, improve data access and agility, enable self-service capabilities, and achieve cost efficiencies. These benefits collectively contribute to making data more accessible, empowering decision-makers and improving overall service delivery in the public sector.
Minh Nguyen is the Regional Director at Denodo in ANZ looking after Australian Federal Government and WA Major Enterprises. He has worked in market leading global organisations such as Qlik, Qualtrics and Snowflake with a strong track record in helping organisations realise the value from their significant investment in data. Minh is currently in his final year of completing his Executive Certificate in Digital Transformation at Harvard Business School.
