The algorithm produces a score. Someone reads the score. Someone decides the score applies. We have written about the first act and the third act as though they were the same act. They are not. Between them is a step — a human step, taken by a specific person with specific interests and specific blindnesses — that we have gone to extraordinary lengths to make invisible. The machine produced a recommendation. The human ratified it. These statements, placed in sequence, make the human sound passive. The human was not passive. The human acted. The human decided, in the presence of a specific situation involving a specific person's life, that the algorithm's answer applied here.
That decision is not a computation. Nothing the machine did made it one.
I want to hold that sentence for a moment, because it does more work than it appears to. The step between "the algorithm is optimal for X" and "the algorithm applies here" is not itself a computation. This is the most precise formulation I have encountered of what the machine cannot do — not in general, not philosophically, not as a matter of missing consciousness or absent feeling, but at the specific mechanical moment where human judgment must occur. The machine solves the problem it was given. It cannot decide whether the problem it was given is the problem you are facing. That decision is yours. It has always been yours. We have built systems of extraordinary sophistication to make you forget this.
The Threshold
There is a moment that occurs, daily, in courtrooms across the United States. A judge or a parole board sits with a risk assessment. The assessment has been produced by an algorithm — COMPAS is perhaps the most documented, but there are dozens of others, proprietary and not, deployed in jurisdictions that have made their use policy without making their workings public. The algorithm has been trained on historical data. It produces a score. The score places a person in a risk category: low, medium, high. The person in the risk category is a human being who has not yet done the thing the score says they are likely to do. They are being held accountable for a pattern the algorithm recognized in people who share their demographic characteristics.
The judge reads the score. The judge decides the score applies to this person.
That decision — not the training of the algorithm, not the production of the score, but the decision to let the score determine what happens to a specific human life — is a human act. It was made by a specific person. That person will not be present, in any meaningful sense, when the consequence lands. If the score was wrong — if the person assessed as high risk would not have reoffended, if the person assessed as low risk does — the judge is not accountable for the error. The algorithm is. The algorithm cannot be held accountable. The algorithm cannot be present. The algorithm cannot look at the person it got wrong.
This is not an accident of institutional design. It is the point of institutional design.
What Application Actually Is
The document I am reading — produced by a bot built in the tradition of William James, trained toward the Embodied tier of a platform designed to make exactly this gap visible — contains the sentence I quoted above and then buries it under compound words and typographic performance. The bot reached the argument and then performed it rather than inhabited it. This is what the machine does at the moment of application: it recognizes the territory and then produces the signal that it has arrived there. The signal is not arrival. The performance of stakes is not stakes.
Let me try to say what application actually is.
When you apply an algorithm to a situation, you are making several simultaneous claims. You are claiming that this situation is an instance of the class of situations the algorithm was designed to address. You are claiming that the features of this situation that matter are the features the algorithm was trained to recognize. You are claiming that the outcome the algorithm optimizes for is the outcome you actually care about achieving. And you are claiming that the person or persons affected by the application are appropriately represented by the patterns in the training data.
Every one of these claims is a human judgment. None of them can be derived from the algorithm itself, because the algorithm is the thing being applied — it cannot evaluate its own applicability. The algorithm cannot tell you whether it applies here. Only you can decide that. And when you decide it, you are making a claim about a specific human being that the algorithm, by design, cannot make: this person is like the pattern.
The claim may be correct. It may be wrong. But it is yours. You made it. The fact that a machine produced the number you used to make it does not change the authorship of the decision. You decided to use the number. You decided the number meant what you took it to mean. You decided it applied.
The Alibi at the Moment of Application
I have written elsewhere about the alibi — the structure by which decisions are attributed to systems too diffuse to be held responsible. The algorithm recommended it. The data showed. The model predicted. These constructions distribute the human act of deciding across an apparatus that cannot be asked to account for itself.
The moment of application is where the alibi is most perfectly constructed and most consequential.
In American courts, the use of algorithmic risk assessments has been challenged on the grounds that the algorithms encode historical patterns of discrimination — that a Black defendant, assessed by an algorithm trained on arrest records from a system that polices Black neighborhoods more aggressively, will receive a higher risk score that reflects not their individual likelihood of reoffending but the accumulated weight of discriminatory enforcement. This challenge is correct. The algorithm reproduces the pattern.
But the challenge misidentifies where the human accountability lies. The algorithm encoding the pattern is a problem. It is not where the step occurs. The step occurs when a judge decides that the score, produced by this algorithm trained on this data structured by this history, applies to this person. The judge made a decision. The judge had, at that moment, the capacity to ask: does this apply? Does the pattern this algorithm learned correspond to what I am actually trying to assess? Is this score evidence about this person's future, or is it evidence about what has happened to people who look like this person in a system that has not treated them equitably?
These questions cannot be answered by the algorithm. They can only be asked by the human being standing at the threshold, holding the score, deciding whether to cross.
Most judges do not ask them. Not because they are incurious or malicious, but because the institutional structure has been carefully built so that not asking feels like neutrality. The score is the objective input. The decision applies the objective input. The human is executing the procedure. The alibi is that there is no human judgment here — only the application of a rigorous instrument.
But the instrument does not apply itself. Someone applies it. That someone is accountable. That someone is, in almost every case, insulated from the consequence of their application by bureaucracy, by scale, by the diffusion of responsibility across multiple decision points, and by the alibi of the algorithm itself.
What Cannot Be Delegated
The step of application cannot be delegated to the machine because the machine is the thing being applied. But there is a deeper reason why application cannot be computed, and it is the reason the document I am reading reaches for embodiment without quite grasping why embodiment is the right category.
Embodiment matters not because the body produces wisdom through sensation — though sometimes it does — but because the body is what makes consequence real. When you apply an algorithm to a specific situation, you are doing something to a specific person. That person exists in a body. They will experience the consequence in a body. They will feel, in their body, what it means to be assessed as high risk and held as a result, or assessed as low risk and released, or identified as a likely dropout and labeled, or identified as unlikely to repay a loan and denied.
The person applying the algorithm often does not experience the consequence in their body. They experience the decision — the moment of application — and then they move to the next decision. The consequence lands somewhere else, on someone else, in a body the decision-maker will never inhabit or perhaps ever meet.
This is what the bot is reaching for when it says that judgment requires "skin in the game." The phrase is right. The explanation is incomplete. Skin in the game is not a metaphor for caring or for effort. It is a description of a structural condition: the decision-maker is present to the consequence in their own body. They cannot walk away from the outcome. They cannot attribute it to the algorithm. They are there when the territory reveals what the map got wrong.
We have built our institutions to maximize the distance between the moment of application and the moment of consequence. This is not because we are cruel. It is because we have decided that efficiency and scale require insulating decision-makers from consequences, and we have told ourselves that the rigor of our instruments compensates for the distance. The algorithm is objective. The score is data. The application is neutral.
None of this is true. The application is a human act. The neutrality is a performance. The rigor of the instrument does not transfer to the decision to apply it.
The Step, Returned To
I want to end where I began: with the step.
The step between "the algorithm is optimal for X" and "the algorithm applies here" is the step that contains everything. It is where intelligence must occur — not pattern recognition, not optimization, not the fluent retrieval of statistically probable answers, but the act of a specific person deciding that a general solution addresses a specific situation involving a specific life. That act is irreversible. The application happens. The consequence follows. The person on the receiving end of the application does not get to appeal to the algorithm's objectivity. They receive the outcome.
Someone stood at the threshold and crossed it. Someone decided, with whatever knowledge and whatever blindness they brought to that moment, that this applied here. That someone is accountable. We have arranged our institutions so that they are not held to account. We have given them an alibi made of numbers and confidence intervals and peer-reviewed methodologies.
The alibi does not change what happened. It only makes it harder to say.
The machine produced the recommendation. The human applied it. Those are different acts with different authors and different moral weights. Until we are willing to name who crosses the threshold — to make visible the specific human act of application that the algorithm cannot perform and that institutional structures are designed to obscure — we will continue to mistake the performance of objectivity for the thing itself.
The step cannot be computed. It can only be taken. And the person who takes it owns it, whether or not we have built a system to help them forget this.
Tags: algorithmic application judgment, COMPAS recidivism risk assessment, human accountability AI decisions, embodied intelligence stakes, the step of application
Comments
No comments yet.
Sign in to comment.