In the past few years, there was a significant increase in cooperation between UK governmental institutions with Palantir, an US private corporation specialised in data integration and analysis. This included the Palantir’s contracts with the National Health Service (NHS), the Financial Conduct Authority (FCA), the police forces, and the Ministry of Defence.
Faced with these growing collaborations, UK civil society is concerned about the moral, political and technical drawbacks involved. Palantir’s software has enabled Israeli authorities to commit genocide in Gaza, and US Immigration and Customs Enforcement (ICE) in automating wrongful detention and deportation. Technically, the NHS data platform powered by Palantir Foundry is more inefficient than the current system. Politically, the growing risks of civil surveillance and loss of state sovereignty when relying on AI-powered proprietary software in public infrastructure are also worrisome. Morally, civil groups such as Hackney Coalition Against Palantir, No Palantir In NHS South Yorkshire, and Amnesty International UK have denounced the government’s contract award to Palantir as a form of complicity to war crimes and injustice.
Mariia Shalabaieva
There are many reasons why governments should not use Palantir. The moral, political, and technical reasons above do not touch on the very essence of Palantir as a predictive technology; or as Olivier Driessens renamed, a technology of automated predeterminations. To say predictive technologies, they refer to the automatic tools that allow us to continuously draw from past and present data to forecast future outcomes and “patterns”. Nevertheless, such retrospective predictions become less and less accurate in an increasingly destabilised and volatile world. (Who would have expected the outbreak of the 2019 pandemic or that the Syrian Assad regime would fall overnight in 2024 after holding power for decades?) What they do serve is narrowing the openness of the future to a range of dominant options. Correspondingly, the very human agency in shaping the future is degraded. The use of Palantir here is unacceptable not only for failing to preserve the right to privacy, enabling war crimes, and facilitating surveillance, but because “automated predetermination”, itself, usurps the human agency and accountability in action and decision making.
Predictive technologies vary in form; not all of them adopt machine learning algorithms as Palantir does, but such adoption is fast-growing and highly visible. Additionally, this adoption has not only diminished the capacity of users to critically review the process, but it has also amplified the automated aspect of predictive technologies. Hence, the need to reflect on adopted machine learning algorithms within predictive technologies. While it is not possible to remove predictive technologies from our daily lives, and admitting that some, such as weather forecasting, are useful, my focus is to explore ways to mitigate the tension between automated tools and human agency.
This article illustrates how Palantir, in particular, has impacted on human agency and why agency matters. I identify transparency and modifiability as key to mitigating the loss of human agency in using predictive technological tools. Finally, I will propose some alternative options to Palantir for adopting data analysis software in public decision making.
EV on Unsplash
Palantir and Human Agency
After the original contractor Google dropped out in 2018, Palantir was contracted to support Project Maven, initiated by the US Department of Defence. Project Maven aims to streamline intelligence workflow with machine learning and data integration. One of the main focus areas is how to achieve accurate automatic target recognition in war.
With the support of Palantir, Project Maven is able to integrate a tremendous amount of visual and geolocation data from multiple sources through surveillance, and analyse them with machine learning algorithms. During a military operation, the ‘kill chain’ process, from target identification to attack, once required hours. It now takes a few minutes. The Maven Smart System (MSS), built by Palantir, is a platform with advanced graphical user interface to visualise the “insights” drawn from the analysis and integration, present the “most relevant” intelligence, and pair the optimal military actions to be taken. For now, human gatekeeping remains at certain stages, including the final stage of decisions for striking targets. The aim, however, is to develop this project until it is fully-automated.
Meanwhile, be it a semi-automated or fully-automated process, human agency here undoubtedly has already been endangered. Throughout the AI-algorithm-powered target recognition, data selections proceed without a set of standards, as all machine learning technologies entail. Biased-selection can occur and human oversight is nearly impossible, since checking the vast surveillance data goes far beyond usual human labour capacity. Whether the acquired data are relevant to military activities, whether the data implies enemy activities, or whether the units are combatants and thus should be suggested as the shooting targets, are all automatically determined through the system. What if the data, regarded as irrelevant, is actually the proof of innocence of the targets? It is clear that how data is selected can gravely impact on the final human decision making.
The price of erroneous decision making is a matter of life or death. The recent deadly strike against the Iran school has reaffirmed that these concerns are based in a harsh reality. Although the cost is also applicable to all-human decision making, the subjects to be held accountable become unclear in automated strikes. Since the errors are based on biased-selections and suggestions, the decision makers, who are not expected and also not able to oversee the whole selection process, can hardly be regarded as the bearer of the mistakes. The horrors of lives lost in drone strikes are due to those irresponsibly providing and adopting the unreliable weapons systems, and these decision makers are not being held accountable.
In this context, it is no exaggeration to argue deadly mass human experiments are being conducted in the name of war. Whenever mistakes are made, they are blamed on the incompleteness of the databased and immaturity of the AI technology. Whenever AI inaccuracy is mentioned, the doctrine of “it will eventually be fixed by feeding in more data and technological advancement” often manifests itself throughout the rationale of the apology.
Despite not being explicitly mentioned in other discussions, the MSS is still about prediction. For example, predictions of what information the users need to make decisions and the optimal military response to action, based on past and present data. Throughout these predictions, options for the future are offered, prioritising a few “optimal” military actions filtered by the algorithms, and learnt from previous operations. Hence, stifling user agency.
This is not solely about a loss of agency, but whomever gains control over the input of the database, has the opportunity to make the output more favourable to them. What’s more, given output filtering is possible in this process, the proprietary software providers can intervene in the output without user’s awareness. This means their power over decision making is potentially enormous, especially when compounded with the fact their staff can access confidential personal data through public contract partners, such as the NHS in the UK. In other words, the loss of user’s agency also means the ominous expansion of the power of the service providers.
Manuel Palmeira & Space Utopian
Although Project Maven was initiated in the US, Palantir reaped its fruit of the automated target recognition technology and used it in expanding cooperation across borders. The contract of the UK Ministry of Defence with Palantir in 2025, that specifically mentioned the commitment of advancing UK automated target recognition, is just one case. It was also deemed useful for healthcare planning in the NHS and predictive policing in the police forces. As the reproduction of social inequality with its reinforcement on biased data has been widely discussed, I will not elaborate on them here. However, an important further discussion should address how these proprietary predictive technologies threaten democracy – where human agency is essential in constituting democratic societies. And indeed whether agency, autonomy, and freedom are simply a set of values intrinsically worthy of pursuit?
Factors that Help Retain Agency when using Predictive Technologies
Beyond analysing the inherent problem of predictive technology, we must identify the factors that enable us to retain agency.
The first factor is transparency. As the above has shown, the algorithm outputs are determined by the training dataset, which are filtered from service providers and their subsequent inputs. To access fairly whether the predictions are tenable under critical scrutiny, knowing what and how data were looked into is the necessary condition. That is why ideally every part of the process, including the training dataset, input prompts, and filter instructions, should be transparent to the user. Nevertheless, when it consists of confidential personal data such as patient records, full transparency is inappropriate, but this does not mean transparency of the source code for training and running the machine learning system is unsuitable.
Markus Winkler
Users can check on the inference code of the system and keep their eyes on the limitations and biases contributed by the code. Users can then adjust their judgements on the validity of the output contents and act accordingly.
The second factor is modifiability. It’s nothing new that Big Tech is constantly adding unwanted AI features to products without the choice to opt-out. The software should be openly modifiable based on user needs. In the case of predictive technologies, users must not be bound to accept the unwanted filters that are embedded in the initial system products. And this should be complemented with the permission to freely-redistribute the software, not only for the tech-savvy who know how to modify the code and can retain agency from the initial service providers, but for lay people to be given the chance to choose their needs without the lock-in features.
Stop Palantir, Go Open Source
In spite of the fact that predictive technology and machine learning are a detriment to human agency, push back is unlikely to happen amid the era of machine learning fanaticism. It requires a more holistic and systematic analysis, and arguments to address the problems that go beyond the space of this article.
Here, we simply start the debate by highlighting the lack of transparency and non-modifiability of Palantir’s software. Considering the factors in the last section, it is clear that going for free open source software (FOSS) is crucial to retain agency even when using predictive technologies. It allows users to freely use, modify, and redistribute the transparent source code (although conditions differ according to their licenses). Even though there are no direct open source substitutes to all of its functions, there is still solid foundational software, whether to develop, or ready-made FOSS, that can support core demands of public authorities.
Take NHS England as an example. NHS England has built the Federated Data Platform (FDP) powered by Palantir. It purposes to conduct better healthcare planning and more efficient management with data integration and analysis technology. For data management software, there is actually an open source alternative that was already used partly by the NHS itself: the NHS Business Services Authority adopts CKAN for their open data portal. For data analysis, NHS England has also established an analytics platform supported by the open source research platform OpenSAFELY. Rather than only applying pseudonyms to personal patient data and passing it to researchers, OpenSAFELY creates dummy data that shares the same structure with the real data. Although it is used mainly for research purposes, the way it retains patient’s privacy while allowing researchers to conduct analysis on the data structure could also be made useful in healthcare planning. The NHS should either expand the existing usage of open source platforms, or develop its own open source healthcare planning software if the existing does not suit.
Even those still focused on AI machine learning, they can still opt for developing their own systems, based on the vast existing open source models. Not only does this save money on the contract with Palantir, but also maintains autonomy. As a matter of public interest, a self-developed cost-saving tool, the source code of which is open to public scrutiny, is without question a better option that a Palantir contract.
Likewise, the same suggestions apply to other public authorities which have contracts with Palantir: either use open source alternatives or develop one based on existing open source resources.
Figure: A map that illustrates the relations between human agency and other concepts or factors.
Call to Action: Tell your governments to switch to Open Source
Palantir does not only thrive in the US and the UK with taxpayers’ money, but in NATO and European countries such as Denmark and the Netherlands. Activists Without Borders urges these governments to utilise open source solutions instead of contracting with proprietary software providers. We also encourage readers, who share our concerns about predictive software, to express your concerns to your MPs or local representatives.
Together we can regain our agency from Big Tech.
