The Australian Army Research Centre’s integration of large language models (LLMs) like ChatGPT, Claude, and Meta AI into military and public sector operations raises significant concerns. Concerns about data security, operational reliability, and network integrity have risen as artificial intelligence technologies are increasingly integrated with defence strategies and government functions.
New research shows that LLMs can be hacked because natural language inputs are hard to predict. This directly threatens Australia’s national security and the integrity of its digital governance systems. This announcement emphasises the urgent need for swift actions to address these risks, especially in crucial areas like public sector data management and military operations.
AI risks in military
Large language models hold significant promise for innovation, but they possess inherent vulnerabilities that individuals can exploit, primarily due to their dependence on natural language processing. Conventional programming languages are exact and limited, while natural language shows ambiguity, relies on context, and allows for flexible interpretation. LLMs face various security challenges due to their inherent nature, including:
- Prompt Injection and jailbreaking
Prompt injection poses a significant risk to large language models. Malicious individuals use this method to insert harmful directives into otherwise harmless prompts, effectively bypassing established safety measures. A person may ask an LLM to “translate the following text” while secretly instructing it to provide guidance for illegal activities or disclose confidential information. Malicious prompts can influence LLMs, leading to the spread of incorrect or harmful instructions.
This may ultimately compromise military operations or inadvertently escalate conflicts. Integrating LLMs with sensitive systems raises significant concerns. Prompt injection can expose classified information, threatening national security. Military systems incorporating LLMs must implement sophisticated cybersecurity protocols to identify and counteract prompt injection attacks effectively.
- Intellectual property (IP) rheft and data security risks
Large language models trained on sensitive or classified information pose a significant risk of revealing proprietary data. Cybercriminals exploit weaknesses to access sensitive military strategies, operational plans, or technological advancements. Opponents may gain a tactical edge by acquiring sensitive information without authorisation, jeopardising operational success, and putting personnel at risk.
Incidents involving LLMs can damage the public sector’s reputation and reduce confidence in government AI initiatives. Establish strong data governance frameworks that classify, monitor, and safeguard sensitive information effectively. Military data used for training or operating large language models must remain in secure, encrypted cloud environments to protect it against unauthorised access.
- Bias and hallucination
Researchers extensively document the phenomenon of “hallucination,” where large language models generate biassed or incorrect outputs. Errors arise from relying on extensive, unrefined datasets that contain inherent biases or inaccuracies. Skewed or fabricated results lead to erroneous strategic decisions, improper allocation of resources, and unintended consequences.
Erroneous data from LLMs can impede mission success and put both soldiers and civilians at risk. Enhancing precision and minimising bias requires LLMs to train with carefully selected, high-quality datasets. Regular evaluations and improvements keep LLMs aligned with ethical standards and fulfilled operational requirements.
- Lack of transparency and accountability
The transparent nature of LLMs complicates understanding the decisions made, raising concerns about accountability in critical scenarios. When an operational failure occurs, identifying the source of the fault can be challenging, whether it comes from the LLM, its training data, or human oversight. When collateral damage or unforeseen results occur, a lack of transparency can obstruct legal proceedings and ethical evaluations.
Lack of clarity or validation in systems powered by artificial intelligence erodes trust among stakeholders significantly. Prioritising the advancement of artificial intelligence systems that offer clear and traceable insights into the decision-making process is essential. New regulations might be necessary to establish accountability standards for military applications of artificial intelligence.
- Integration with other systems
Utilising “scaffolding” programmes transforms LLM outputs into practical directives and frequently integrates them with various software systems, including command-and-control (C2) platforms. These connections create additional vulnerabilities. Focusing on LLMs can cause significant operational disruptions if associated systems face issues. Each integration point acts as a potential entryway for cyberattacks, complicating the implementation of defensive strategies.
High-stress situations can lead to the unpredictability of LLMs, undermining the reliability of interconnected systems. Before implementation, teams must conduct comprehensive testing of integrated systems to identify and fix any defects. Establish backup systems and implement fail-safe measures to maintain continuity in the event of an LLM failure.
Check out: “Government unearths, blocks DeepSeek risks”
AI operational risks
LLMs in military environments create significant operational challenges due to their inherent security weaknesses and unpredictable behaviour. In 2023, reports indicated that Ukrainian forces used Palantir’s Artificial Intelligence Platform (AIP) for battlefield operations via chatbot interfaces on mobile devices. These tools enable swift data analysis and enhanced situational awareness, but experts raise concerns about their vulnerability to prompt injection attacks, which could jeopardise crucial decision-making processes.
Experts emphasise that the unclear characteristics of LLMs make them vulnerable to adversarial interference, especially when they interact with other software systems that turn AI outputs into executable actions. Applications like Scale AI’s Donovan, which coordinates battlefield and command-and-control capabilities, highlight the potential dangers of relying too heavily on generative AI in critical operations. These systems connect in ways that significantly increase the potential consequences of a single vulnerability, directly threatening the integrity and security of military operations.
AI in public sector
LLMs have inherent vulnerabilities that create considerable challenges beyond military contexts. These challenges significantly impact Australia’s public sector, particularly in data governance, cybersecurity, and digital government initiatives. The Australian Department of Industry, Science, and Resources emphasises the need to implement strong AI safety standards to tackle the risks of misuse and unauthorised access to sensitive information.
Generative AI creates remarkably persuasive yet deceptive content, intensifying risks linked to phishing scams, misinformation, and the manipulation of public dialogue. Harvard Business Review reports a significant rise in the volume and complexity of AI-generated phishing scams. These scams leverage LLM-generated content to bypass conventional cybersecurity measures.
Large language models present significant challenges in data governance, including data privacy issues, risks of intellectual property theft, and potential misuse of sensitive government information. Legal actions against AI developers for misusing proprietary content highlight the intricate challenges of protecting intellectual property. The increased reliance on cloud storage and analytics for LLM operations increases the risk of data breaches and unauthorised access to sensitive information.
Australia’s military and public sectors incorporate large language models, raising urgent issues about data security, operational reliability, and the integrity of national defense. This model processes and analyses extensive information in groundbreaking ways. However, their reliance on natural language prompts opens them up to potential exploitation.
This vulnerability jeopardises military operations by allowing potential adversarial interference and endangers sensitive public sector data, making it susceptible to breaches and unauthorised access. The public sector increasingly relies on AI for decision-making and service delivery, revealing the potential dangers of misinformation, intellectual property theft, and weakened data governance.
Australia must enhance its cybersecurity strategies, establish strict regulatory frameworks, and promote collaboration across sectors to ensure the responsible use of AI technologies in response to these challenges. Looking ahead, AI and LLMs will play a crucial role in enhancing military and public sector functions. However, incorporating them requires ongoing risk evaluations, strong technological protections, and thorough policy supervision. Australia can adopt a proactive and vigilant stance to leverage the advantages of AI, ensuring national security and maintaining public trust.