O nooscópio

Fichamento do recente trabalho The Nooscope Manifested: Artificial Intelligence as Instrument of Knowledge Extractivism^[1] de Matteo Pasquinelli e Vladan Joler, que acompanha um interessante diagrama disponível no site do trabalho (nooscope.ai).

O projeto de mecanizar a razão

Sobre os termos do debate

O nooscópio como cartografia e manifesto (perspectiva parcial).

The Nooscope is a cartography of the limits of artificial intelligence, intended as a provocation to both computer science and the humanities. Any map is a partial perspective, a way to provoke debate. Similarly, this map is a manifesto — of AI dissidents. (p. 2)

Inteligência artificial e autonomia

Sobre o termo "inteligência artificial" e o mito que ele sustenta. Autonomia de inteligência (diferente da inteligência natural) e como projeto político (separado da humanidade).

Its main purpose is to challenge the mystifications of artificial intelligence. First, as a technical definition of intelligence and, second, as a political form that would be autonomous from society and the human. (p. 2)

O mito da autonomia tecnológica (neutralidade da ciência, determinismo tecnológico). Encoberta 2 processos de alienação: autonomia das corporações produtoras de tecnologia de ponta; invisibilização da autonomia dos trabalhadores.

In the expression ‘artificial intelligence’ the adjective ‘artificial’ carries the myth of the technology’s autonomy: it hints to caricatural ‘alien minds’ that self-reproduce in silico but, actually, mystifies two processes of proper alienation: the growing geopolitical autonomy of hi-tech companies and the invisibilization of workers’ autonomy worldwide. (p. 2)

Aprendizado de máquina e o regime de extrativismo cognitivo e colonialismo epistêmico

Projeto de mecanização da razão no séc XXI, cujo objetivo é o extrativismo de conhecimento e o colonialismo epistêmico. Aprendizado de máquina como compressão de informação.

The modern project to mechanise human reason has clearly mutated, in the 21 st century, into a corporate regime of knowledge extractivism and epistemic colonialism. This is unsurprising, since machine learning algorithms are the most powerful algorithms for information compression. (p. 2)

Secularizar a IA: instrumento de navegação pelo conhecimento

The purpose of the Nooscope map is to secularize AI from the ideological status of ‘intelligent machine’ to one of knowledge instrument. Rather than evoking legends of alien cognition, it is more reasonable to consider machine learning as an instrument of knowledge magnification that helps to perceive features, patterns, and correlations through vast spaces of data beyond human reach. In the history of science and technology, this is no news: it has already been pursued by optical instruments throughout the histories of astronomy and medicine. In the tradition of science, machine learning is just a Nooscope, an instrument to see and navigate the space of knowledge (from the Greek skopein ‘to examine, look’ and noos ‘knowledge’). (p. 2)

O nooscópio de Leibniz como instrumento de amplificação.

Discussing the power of his calculus ratiocinator and ‘characteristic numbers’ (the idea to design a numerical universal language to codify and solve all the problems of human reasoning), Leibniz made an analogy with instruments of visual magnification such as the microscope and telescope. He wrote: ‘Once the characteristic numbers are established for most concepts, mankind will then possess a new instrument which will enhance the capabilities of the mind to a far greater extent than optical instruments strengthen the eyes, and will supersede the microscope and telescope to the same extent that reason is superior to eyesight.’ (p. 2)

Imperfeições do nooscópio

Como qualquer instrumento de medida/percepção, o nooscópio possui erros, viéses e imperfeições.

Instruments of measurement and perception always come with inbuilt aberrations. In the same way that the lenses of microscopes and telescopes are never perfectly curvilinear and smooth, the logical lenses of machine learning embody faults and biases. (p. 2)

Para além do viés em aprendizado de máquina. Episteme de causa x Episteme correlação.

To understand machine learning and register its impact on society is to study the degree by which social data are diffracted and distorted by these lenses. This is generally known as the debate on bias in AI, but the political implications of the logical form of machine learning are deeper. Machine learning is not bringing a new dark age but one of diffracted rationality, in which, as it will be shown, an episteme of causation is replaced by one of automated correlations. (p. 2)

IA como novo regime de verdade, o de correlações estatísticas automatizadas:

More in general, AI is a new regime of truth, scientific proof, social normativity and rationality, which often does take the shape of a statistical hallucination. This diagram manifesto is another way to say that AI, the king of computation (patriarchal fantasy of mechanised knowledge, ‘master algorithm’ and alpha machine) is naked. Here, we are peeping into its black box. (p. 2)

Linha de montagem do aprendizado de máquina: Dado, Algoritmo, Modelo

IA como prática sem teoria

Although corporate AI today describes its power with the language of ‘black magic’ and ‘superhuman cognition’, current techniques are still at the experimental stage. AI is now at the same stage as when the steam engine was invented, before the laws of thermodynamics necessary to explain and control its inner workings, had been discovered. Similarly, today, there are efficient neural networks for image recognition, but there is no theory of learning to explain why they work so well and how they fail so badly. Like any invention, the paradigm of machine learning consolidated slowly, in this case through the last half-century. (p. 3)

Como instrumento de conhecimento, aprendizado de máquina é separado em objeto observável (base de dados de treinamento), um instrumento de observação (algoritmo de aprendizado) e uma representação final (modelo estatístico).

Staying with the analogy of optical media, the information flow of machine learning is like a light beam that is projected by the training data, compressed by the algorithm and diffracted towards the world by the lens of the statistical model. (p. 3)

IA não possui apenas viés técnico - por vezes negado em uma forma de fetiche das tecnologias baseados em dado e estatística - mas também viés humano:

This double operation [estudar como funciona e onde falha] stresses that AI is not a monolithic paradigm of rationality but a spurious architecture made of adapting techniques and tricks. Besides, the limits of AI are not simply technical but are imbricated with human bias. In the Nooscope diagram the essential components of machine learning are represented at the centre, human biases and interventions on the left, and technical biases and limitations on the right. Optical lenses symbolize biases and approximations representing the compression and distortion of the information flow. The total bias of machine learning is represented by the central lens of the statistical model through which the perception of the world is diffracted. (p. 4)

O problema do viés

The limitations of AI are generally perceived today thanks to the discourse on bias —the amplification of gender, race, ability, and class discrimination by algorithms. In machine learning, it is necessary to distinguish between historical bias, dataset bias, and algorithm bias, all of which occur at different stages of the information flow. (p. 4)

Viés histórico. Este viés existe antes das tecnologias de IA, mas é potencializado pelo discurso de neutralidade acerca destas.

Historical bias (or world bias) is already apparent in society before technological intervention. Nonetheless, the naturalisation of such bias, that is the silent integration of inequality into an apparently neutral technology is by itself harmful. Paraphrasing Michelle Alexander, Ruha Benjamin has called it the New Jim Code: ‘the employment of new technologies that reflect and reproduce existing inequalities but that are promoted and perceived as more objective or progressive than the discriminatory systems of a previous era.’ (p. 4)

Viés de base de dados (por exemplo, no processo de rotulação). Conferir [2].

Dataset bias is introduced through the preparation of training data by human operators. The most delicate part of the process is data labelling, in which old and conservative taxonomies can cause a distorted view of the world, misrepresenting social diversities and exacerbating social hierarchies (see below the case of ImageNet). (p. 4)

Viés algorítmico, viés estatístico, viés de modelo (e o problema de compressão de informação):

Algorithmic bias (also known as machine bias, statistical bias or model bias, to which the Nooscope diagram gives particular attention) is the further amplification of historical bias and dataset bias by machine learning algorithms. The problem of bias has mostly originated from the fact that machine learning algorithms are among the most efficient for information compression, which engenders issues of information resolution, diffraction and loss. (p. 4)

O problema da compressão de informação

Em prol da aparente eficiência de recursos - o que é no mínimo questionável dado a quantidade massiva de energia, tempo, espaço e trabalho organizados para sustentar as tecnologias atuais de IA - e da velha conhecida acumulação de capital (através da capitalização de dados), o processo de compressão de informação produz discriminação e perda de diversidade cultural.

Since ancient times, algorithms have been procedures of an economic nature, designed to achieve a result in the shortest number of steps consuming the least amount of resources: space, time, energy and labour. 13 The arms race of AI companies is, still today, concerned with finding the simplest and fastest algorithms with which to capitalise data. If information compression produces the maximum rate of profit in corporate AI, from the societal point of view, it produces discrimination and the loss of cultural diversity. (p. 4)

O problema da caixa preta

Apesar de desenvovimentos em algoritmos de interpretabilidade (como análise de sensibilidade ou Layer-wise Relevance Analysis), um problema da IA moderna é sua opacidade, pelo menos em termos de funcionamento ("o que o algoritmo está fazendo?"). Isso abre margem para um discurso de que estes seriam não só impossíveis de serem estudados, mas também de serem controlados.

While the social consequences of AI are popularly understood under the issue of bias, the common understanding of technical limitations is known as the black box problem. The black box effect is an actual issue of deep neural networks (which filter information so much that their chain of reasoning cannot be reversed) but has become a generic pretext for the opinion that AI systems are not just inscrutable and opaque, but even ‘alien’ and out of control. 14 The black box effect is part of the nature of any experimental machine at the early stage of development (it has already been noticed that the functioning of the steam engine remained a mystery for some time, even after having been successfully tested). The actual problem is the black box rhetoric, which is closely tied to conspiracy theory sentiments in which AI is an occult power that cannot be studied, known, or politically controlled. (p. 4)

Base de dados: a origem social da inteligência maquínica

Capitalismo de vigilância [3].

Mass digitalisation, which expanded with the Internet in the 1990s and escalated with datacentres in the 2000s, has made available vast resources of data that, for the first time in history, are free and unregulated. A regime of knowledge extractivism (then known as Big Data) gradually employed efficient algorithms to extract ‘intelligence’ from these open sources of data, mainly for the purpose of predicting consumer behaviours and selling ads. The knowledge economy morphed into a novel form of capitalism, called cognitive capitalism and then surveillance capitalism, by different authors. It was the Internet information overflow, vast datacentres, faster microprocessors and algorithms for data compression that laid the groundwork for the rise of AI monopolies in the 21st century. (p. 5)

Inteligência da IA moderna é dependente da base de dados (IA moderna é composta em sua grande maioria por algoritmos adaptativos baseados em dado - aprendizado de máquina - sendo grande parte deles redes neurais e aprendizado profundo).

The quality of training data is the most important factor affecting the so-called ‘intelligence’ that machine learning algorithms extract. There is an important perspective to take into account, in order to understand AI as a Nooscope. Data is the first source of value and intelligence. (p. 5)

Construção social dos dados

Dado "bruto", nunca é neutro [4].

However, training data are never raw, independent and unbiased (they are already themselves ‘algorithmic’). (p. 5)

The act of selecting one data source rather than another is the profound mark of human intervention into the domain of the ‘artificial’ minds. (p. 5)

Metadado (que é, por si só, dado) geralmente é usado como categorizações discretizadas e ideais.

The training dataset is a cultural construct, not just a technical one. It usually comprises input data that are associated with ideal output data, such as pictures with their descriptions, also called labels or metadata. (p. 5)

A realidade (analógica) é sempre mais complexa que um computador (digital) pode expressar.

The semiotic process of assigning a name or a category to a picture is never impartial; this action leaves another deep human imprint on the final result of machine cognition. (p. 5)

Produção-captura-rotulação:

A training dataset for machine learning is usually composed through the following steps: 1) production: labour or phenomena that produce information; 2) capture: encoding of information into a data format by an instrument: 3) formatting: organisation of data into a dataset: 4) labelling: in supervised learning, the classification of data into categories (metadata).

Bases de dados não são nem tecnica nem socialmente imparciais.

Machine intelligence is trained on vast datasets that are accumulated in ways neither technically neutral nor socially impartial. Raw data does not exist, as it is dependent on human labour, personal data, and social behaviours that accrue over long periods, through extended networks and controversial taxonomies. (p. 5)

Colonialismo e divisão de trabalho na manufatura de dados

The main training datasets for machine learning (NMIST, ImageNet, Labelled Faces in the Wild, etc.) originated in corporations, universities, and military agencies of the Global North. But taking a more careful look, one discovers a profound division of labour that innervates into the Global South via crowdsourcing platforms that are used to edit and validate data. (p. 5-6)

Sobre o ImageNet:

At the end of the day (and of the assembly line), anonymous workers from all over the planet were paid few cents per task to label hundreds of pictures per minute according to the WordNet taxonomy: their labour resulted in the engineering of a controversial cultural construct. (p. 6)

Ao passo que militantes, ativistas e outraos interessadaos em compartilhamento de informação e liberdade de informação por meio de movimentos como o Creative Commons, uma quantidade enorme de imagens e dados foi apropriada para alimentar o mercado de IA:

The voracious data extractivism of AI has caused an unforeseeable backlash on digital culture: in the early 2000s, Lawrence Lessig could not predict that the large repository of online images credited by Creative Commons licenses would a decade later become an unregulated resource for face recognition surveillance technologies. (p. 6)

Bases de dado e privacidade

In similar ways, personal data is continually incorporated without transparency into privatised datasets for machine learning. (p. 6)

Online training datasets trigger issues of data sovereignty and civil rights that traditional institutions are slow to counteract (see the European General Data Protection Regulation). (p. 6)

If 2012 was the year in which the Deep Learning revolution began, 2019 was the year in which its sources were discovered to be vulnerable and corrupted. (p. 6)

História da IA como a automação da percepção

Head of Facebook AI and godfather of convolutional neural networks Yann LeCun reiterates that current AI systems are not sophisticated versions of cognition, but rather, of perception. (p. 7)

Perceptron

Perceptron de Rosenblat, originalmente como uma máquina de reconhecimento de padrões.

The archetype machine for pattern recognition is Frank Rosenblatt’s Perceptron. Invented in 1957 at Cornell Aeronautical Laboratory in Buffalo, New York, its name is a shorthand for ‘Perceiving and Recognizing Automaton.’ 26 Given a visual matrix of 20x20 photoreceptors, the Perceptron can learn how to recognise simple letters. A visual pattern is recorded as an impression on a network of artificial neurons that are firing up in concert with the repetition of similar images and activating one single output neuron. The output neuron fires 1=true, if a given image is recognised, or 0=false, if a given image is not recognised. (p. 7)

McCullochs, Pitts e além

As redes neurais evoluíram como algoritmos de reconhecimento de padrões. Vide [5] sobre aprendizado profundo como aprendizado de representação.

The automation of perception, as a visual montage of pixels along a computational assembly line, was originally implicit McCulloch and Pitt’s concept of artificial neural networks. 27 Once the algorithm for visual pattern recognition survived the ‘winter of AI’ and proved efficient in the late 2000s, it was applied also to non-visual datasets, properly inaugurating the age of Deep Learning (the application of pattern recognition techniques to all kinds of data, not just visual). Today, in the case of self-driving cars, the patterns that need to be recognised are objects in road scenarios. In the case of automatic translation, the patterns that need to be recognised are the most common sequences of words across bilingual texts. (p. 7)

Codificação cultural

Regardless of their complexity, from the numerical perspective of machine learning, notions such as image, movement, form, style, and ethical decision can all be described as statistical distributions of pattern. In this sense, pattern recognition has truly become a new cultural technique that is used in various fields. (p. 7)

Talvez um termo que possa substituir "treinamento" possa ser aprendizado de representação ( representational learning) e além de classificação e predição poderíamos adicionar clusterização e aprendizado por reforço que, embora fujam um pouco das 3 formas anteriores (o primeiro por não exigir uma etapa de rotulação, e o segundo por gerar os próprios dados que o fazem aprender) também lidam com codificação/decodificação de padrões.

For explanatory purposes, the Nooscope is described as a machine that operates on three modalities: training, classification, and prediction. In more intuitive terms, these modalities can be called: pattern extraction, pattern recognition, and pattern generation. (p. 7)

Inferência estatística e modelo estatístico

What a neural network computes, is not an exact pattern but the statistical distribution of a pattern. Just scraping the surface of the anthropomorphic marketing of AI, one finds another technical and cultural object that needs examination: the statistical model. (p. 7)

In terms of the work of demystification that needs to be done (also to evaporate some naïve questions), it would be good to reformulate the trite question ‘Can a machine think?’ into the theoretically sounder questions ‘Can a statistical model think?’, ‘Can a statistical model develop consciousness?' et cetera. (p. 7)

Comprimir o mundo em um modelo estatístico

Algoritmo de IA como fórmula alquímica

The algorithms of AI are often evoked as alchemic formulas, capable of distilling ‘alien’ forms of intelligence. (p. 8)

Opacidade do modelo estatístico

Especificamente em redes neurais artificiais, o modelo estatístico é construído durante o aprendizado, na forma de parâmetros do algoritmo, chamados pesos sinápticos em analogia com as redes neurais biológicas.

Algorithm is the name of a process, whereby a machine performs a calculation. The product of such machine processes is a statistical model (more accurately termed an ‘algorithmic statistical model’). In the developer community, the term ‘algorithm’ is increasingly replaced with ‘model.’ This terminological confusion arises from the fact that the statistical model does not exist separately from the algorithm: somehow, the statistical model exists inside the algorithm under the form of distributed memory across its parameters. (p. 8)

Apesar do trecho abaixo, existem algoritmos de visualização dos parâmetros que aparentemente passam a ser uma preocupação maior no campo (6, 7).

For the same reason, it is essentially impossible to visualise an algorithmic statistical model, as is done with simple mathematical functions. Still, the challenge is worthwhile. (p. 8)

Parâmetros, hiperparâmetros, saída

Modelos estatísticos são controlados por parâmetros discretos. Poucos hiperparâmetros controlam milhões de parâmetros que expressará em sua saída não necessariamente um valor binário (classificação multiclasse), nem um valor discreto (por exemplo, no caso de regressão), mas sempre de uma forma comprimida (real ~ contínuo, modelo ~ discreto).

Artificial neural networks started as simple computing structures that evolved into complex ones which are now controlled by a few hyperparameters that express millions of parameters. 28 For instance, convolutional neural networks are described by a limited set of hyperparameters (number of layers, number of neurons per layer, type of connection, behaviour of neurons, etc.) that project a complex topology of thousands of artificial neurons with millions of parameters in total. The algorithm starts as a blank slate and, during the process called training, or ‘learning from data', adjusts its parameters until it reaches a good representation of the input data. In image recognition, as already seen, the computation of millions of parameters has to resolve into a simple binary output: 1=true, a given image is recognised; or 0=false, a given image is not recognised. (p. 8)

Aprendizado de máquina como inferência estatística automatizada

Statistical models have always influenced culture and politics. They did not just emerge with machine learning: machine learning is just a new way to automate the technique of statistical modelling. (p. 9)

Climate models are historical artefacts that are tested and debated within the scientific community, and today, also beyond. (p. 9)

IA como ficção científica estatística.

Machine learning models, on the contrary, are opaque and inaccessible to community debate. Given the degree of myth-making and social bias around its mathematical constructs, AI has indeed inaugurated the age of statistical science fiction. Nooscope is the projector of this large statistical cinema. (P. 9)

Todos os modelos estão errados, mas alguns são úteis (George Box)

This maxim, however, is often used to legitimise the bias of corporate and state AI. (p. 10)

Mapa versus território

Computer scientists argue that human cognition reflects the capacity to abstract and approximate patterns. So what’s the problem with machines being approximate, and doing the same? Within this argument, it is rhetorically repeated that ‘the map is not the territory’. This sounds reasonable. But what should be contested is that AI is a heavily compressed and distorted map of the territory and that this map, like many forms of automation, is not open to community negotiation. AI is a map of the territory without community access and community consent. (p. 10)

Trabalho de percepção (reconhecimento de padrão)

The algorithm does not know what an image is, does not perceive an image as human cognition does, it only computes pixels, numerical values of brightness and proximity. (p. 10)

Sobreajuste e sobajuste

In this case, the model is overfitting, because it has meticulously learnt everything (including noise) and is not able to distinguish a pattern from its background. On the other hand, the model is underfitting when it is not able to detect meaningful patterns from the training data. (p. 10)

Aprendizado de máquina não aprende nada

De fato algoritmos de aprendizado de máquina estão automatizando um trabalho de indução (generalização a partir de casos locais) baseado em dados e regras estritas. Um algoritimo não pode classificar para além das classes especificadas, ou maximizar algo que não esteja pré-definido na forma de uma função matemática. Mesmo algoritmos de aprendizado não supervisionado não conseguem criar algo novo, mas sim organizar/mapear/representar dados a partir de métricas e regras pré-estabelecidas.

Machine learning is a term that, as much as ‘AI', anthropomorphizes a piece of technology: machine learning learns nothing in the proper sense of the word, as a human does; machine learning simply maps a statistical distribution of numerical values and draws a mathematical function that hopefully approximates human comprehension. That being said, machine learning can, for this reason, cast new light on the ways in which humans comprehend. (p. 11)

The statistical model of machine learning algorithms is also an approximation in the sense that it guesses the missing parts of the data graph: either through interpolation, which is the prediction of an output y within the known interval of the input x in the training dataset, or through extrapolation, which is the prediction of output y beyond the limits of x, often with high risks of inaccuracy. This is what ‘intelligence’ means today within machine intelligence: to extrapolate a non-linear function beyond known data boundaries. As Dan McQuillian aptly puts it: ‘There is no intelligence in artificial intelligence, nor does it learn, even though its technical name is machine learning, it is simply mathematical minimization.’ (p. 11)

Em geral, o termo força bruta é utilizado quando se testam todas as possibilidades sem um critério norteador. Mas de fato, as tecnologias de IA modernas são métodos heurísticos de tentativa e melhora sucessivas.

It is important to recall that the ‘intelligence’ of machine learning is not driven by exact formulas of mathematical analysis, but by algorithms of brute force approximation. The shape of the correlation function between input x and output y is calculated algorithmically, step by step, through tiresome mechanical processes of gradual adjustment (like gradient descent, for instance) that are equivalent to the differential calculus of Leibniz and Newton. (p. 11)

O grande potencial de redes neurais recai (pelo menos de forma necessária, mesmo que não de forma suficiente) sobre a lei de aproximação universal [8].

Neural networks are said to be among the most efficient algorithms because these differential methods can approximate the shape of any function given enough layers of neurons and abundant computing resources. (p. 11)

Pegada de carbono de algoritmos de IA

Brute-force gradual approximation of a function is the core feature of today’s AI, and only from this perspective can one understand its potentialities and limitations — particularly its escalating carbon footprint (the training of deep neural networks requires exorbitant amounts of energy because of gradient descent and similar training algorithms that operate on the basis of continuous infinitesimal adjustments). (p. 11)

World2vec

Conferir [9].

Before being analysed, data are encoded into a multi-dimensional vector space that is far from intuitive. (p. 12)

De fato, a seleção de arquitetura é feito com base em tentativa e erro, e seu sucesso é "constatado" empiricamente.

It must be stressed, however, that machine learning still resembles more craftsmanship than exact mathematics. AI is still a history of hacks and tricks rather than mystical intuitions. (p. 13)

Apesar dos espaços vetoriais resolverem o problema da "maldição da dimensionalidade", não foi exatamente uma compressão da informação, mas sim uma mudança de sua representação. Ao se mudar de espaços discretos para espaços contínuos, a mesma informação pode ser representada de uma forma diferente. Isso seria comprimir a informação em si? Eu não tenho tanta certeza.

For example, one trick of information compression is dimensionality reduction, which is used to avoid the Curse of Dimensionality, that is the exponential growth of the variety of features in the vector space. The dimensions of the categories that show low variance in the vector space (i.e. whose values fluctuate only a little) are aggregated to reduce calculation costs. (p. 13)

O problema parece maior quando a redução de dimensionalidade acontece no próprio conteúdo, por exemplo ao reduzir a complexidade do real à classes discretizadas.

Dimensionality reduction can be used to cluster word meanings (such as in the model word2vec) but can also lead to category reduction, which can have an impact on the representation of social diversity. Dimensionality reduction can shrink taxonomies and introduce bias, further normalising world diversity and obliterating unique identities. (p. 13)

A sociedade da classificação e dos bots de predição

Governabilidade algoritmica

Classificação-predição (reconhecimento-geração) como governabilidade algoritmica [10].

Most of the contemporary applications of machine learning can be described according to the two modalities of classification and prediction, which outline the contours of a new society of control and statistical governance. Classification is known as pattern recognition, while prediction can be defined also as pattern generation. A new pattern is recognised or generated by interrogating the inner core of the statistical model. (p. 13)

Machine learning classification and prediction are becoming ubiquitous techniques that constitute new forms of surveillance and governance. Some apparatuses, such as self-driving vehicles and industrial robots, can be an integration of both modalities. (p. 14)

Isso me faz lembrar dos conceitos de biopolítica, racismo, população e tecnologias de poder presentes em Foucault (11, 12). A distopia que se apresenta também parece uma forma ainda mais intensa do conceito de "sociedade de controle" desenhado por Deleuze em [13].

Even if recognising an obstacle on a road seems to be a neutral gesture (it’s not), identifying a human being according to categories of gender, race and class (and in the recent COVID-19 pandemic as sick or immune), as state institutions are increasingly doing, is the gesture of a new disciplinary regime. (p. 14)

The hubris of automated classification has caused the revival of reactionary Lombrosian techniques that were thought to have been consigned to history, techniques such as Automatic Gender Recognition (AGR), ‘a subfield of facial recognition that aims to algorithmically identify the gender of individuals from photographs or videos.’ (p. 14)

IA e arte

Embora redes neurais sejam capaz de gerar padrões, estes são sempre baseados em um conjunto de dados previamente coletado, de forma que seria estranho atribuir qualquer forma de potencial criativo a uma GAN ou a um algoritmo de regressão.

Recently, the generative modality of machine learning has had a cultural impact: its use in the production of visual artefacts has been received by mass media as the idea that artificial intelligence is ‘creative’ and can autonomously make art. An artwork that is said to be created by AI always hides a human operator, who has applied the generative modality of a neural network trained on a specific dataset. (p. 14)

Sobre a criação de "halucinações" em redes profundas:

In this modality, the neural network is run backwards (moving from the smaller output layer toward the larger input layer) to generate new patterns after being trained at classifying them, a process that usually moves from the larger input layer to the smaller output layer. The generative modality, however, has some useful applications: it can be used as a sort of reality check to reveal what the model has learnt, i.e. to show how the model ‘sees the world.’ It can be applied to the model of a self-driving car, for instance, to check how the road scenario is projected. (p. 14)

Sobre as GANs e seu potencial assustador para criação de fake news e outras formas de fraude:

The two main modalities of classification and generation can be assembled in further architectures such as in the Generative Adversarial Networks. In the GAN architecture, a neural network with the role of discriminator (a traditional classifier) has to recognise an image produced by a neural network with the role of generator, in a reinforcement loop that trains the two statistical models simultaneously. For some converging properties of their respective statistical models, GANs have proved very good at generating highly realistic pictures. This ability has prompted their abuse in the fabrication of ‘deep fakes’. 46 Concerning regimes of truth, a similar controversial application is the use of GANs to generate synthetic data in cancer research, in which neural networks trained on unbalanced datasets of cancer tissues have started to hallucinate cancer where there was none. 47 In this case ‘instead of discovering things, we are inventing things', Fabian Offert notices, ‘the space of discovery is identical to the space of knowledge that the GAN has already had. […] While we think that we are seeing through GAN — looking at something with the help of a GAN — we are actually seeing into a GAN. GAN vision is not augmented reality, it is virtual reality. GANs do blur discovery and invention.’ The GAN simulation of brain cancer is a tragic example of AI-driven scientific hallucination. (p. 15)

As falhas de um instrumento estatístico

Poder de normalização

AI easily extends the ‘power of normalisation’ of modern institutions, among others bureaucracy, medicine and statistics (originally, the numerical knowledge possessed by the state about its population) that passes now into the hands of AI corporations. The institutional norm has become a computational one: the classification of the subject, of bodies and behaviours, seems no longer to be an affair for public registers, but instead for algorithms and datacentres. ‘Data-centric rationality’, Paula Duarte has concluded, ‘should be understood as an expression of the coloniality of power. (p. 15)

O potencial hiperconservador dos algoritmos de aprendizado de máquina parecem ser - e provavelmente não por coincidência - extremamente compatíveis com o desenho que Guattari faz do papel da culturas nas sociedades capitalísticas [14] em subtrair o singular.

If gender, race and class discriminations are amplified by AI algorithms, this is also part of a larger problem of discrimination and normalisation at the logical core of machine learning. The logical and political limitation of AI is the technology’s difficulty in the recognition and prediction of a new event. (p. 16)

A logical limit of machine learning classification, or pattern recognition, is the inability to recognise a unique anomaly that appears for the first time, such as a new metaphor in poetry, a new joke in everyday conversation, or an unusual obstacle (a pedestrian? a plastic bag?) on the road scenario. (p. 16)

As a technique of information compression, machine learning automates the dictatorship of the past, of past taxonomies and behavioural patterns, over the present. This problem can be termed the regeneration of the old — the application of a homogenous space-time view that restrains the possibility of a new historical event. (p. 16)

Aprendizado de máquina e criatividade

Interestingly, in machine learning, the logical definition of a security issue also describes the logical limit of its creative potential. The problems characteristic of the prediction of the new are logically related to those that characterise the generation of the new, because the way a machine learning algorithm predicts a trend on a time chart is identical to the way it generates a new artwork from learnt patterns. The hackneyed question ‘Can AI be creative?’ should be reformulated in technical terms: is machine learning able to create works that are not imitations of the past? Is machine learning able to extrapolate beyond the stylistic boundaries of its training data? The ‘creativity’ of machine learning is limited to the detection of styles from the training data and then random improvisation within these styles. In other words, machine learning can explore and improvise only within the logical boundaries that are set by the training data. For all these issues, and its degree of information compression, it would be more accurate to term machine learning art as statistical art. (p. 16)

Correlação não implica causa-consequência

Another unspoken bug of machine learning is that the statistical correlation between two phenomena is often adopted to explain causation from one to the other. In statistics, it is commonly understood that correlation does not imply causation, meaning that a statistical coincidence alone is not sufficient to demonstrate causation. (p. 16)

Isso seria menos um problema se o debate acerca do campo de IA fosse transparente e os algoritmos fossem vistos como modelos estatísticos que indicam correlações, mas não é o que acontece.

Superficially mining data, machine learning can construct any arbitrary correlation that is then perceived as real. (p. 16)

IA como aparato biopolítico preemptivo

Such a logical fallacy has already become a political one, if one considers that police forces worldwide have adopted predictive policing algorithms. (p. 16)

According to Dan McQuillan, when machine learning is applied to society in this way, it turns into a biopolitical apparatus of preemption, that produces subjectivities which can subsequently be criminalized. (p. 16)

Ultimately, machine learning obsessed with ‘curve fitting’ imposes a statistical culture and replaces the traditional episteme of causation (and political accountability) with one of correlations blindly driven by the automation of decision making. (p. 16)

Inteligência adversarial versus inteligência artificial

Sujeito constrói o objeto

How to emancipate ourselves from a data-centric view of the world? It is time to realise that it is not the statistical model that constructs the subject, but rather the subject that structures the statistical model. (p. 17)

Internalist and externalist studies of AI have to blur: subjectivities make the mathematics of control from within, not from without. To second what Guattari once said of machines in general, machine intelligence too is constituted of ‘hyper-developed and hyper- concentrated forms of certain aspects of human subjectivity.’ (p. 17)

Hacking

Hacking is an important method of knowledge production, a crucial epistemic probe into the obscurity of AI. (p. 17)

Ataques adversariais

Adversarial attacks exploit blind spots and weak regions in the statistical model of a neural network, usually to fool a classifier and make it perceive something that is not there. (p. 18)

Envenenamento de dado

This effect is achieved also by reverse-engineering the statistical model or by polluting the training dataset. In this latter sense, the technique of data poisoning targets the training dataset and introduces doctored data. In so doing it alters the accuracy of the statistical model and creates a backdoor that can be eventually exploited by an adversarial attack. (p. 18)

Vulnerabilidades

Adversarial attack seems to point to a mathematical vulnerability that is common to all machine learning models: ‘An intriguing aspect of adversarial examples is that an example generated for one model is often misclassified by other models, even when they have different architectures or were trained on disjoint training sets.’ (p. 19)

Anomalo como posição política

Adversarial attacks remind us of the discrepancy between human and machine perception and that the logical limit of machine learning is also a political one. (p. 19)

The logical and ontological boundary of machine learning is the unruly subject or anomalous event that escapes classification and control. The subject of algorithmic control fires back. Adversarial attacks are a way to sabotage the assembly line of machine learning by inventing a virtual obstacle that can set the control apparatus out of joint. An adversarial example is the sabot in the age of AI. (p. 19)

Trabalho na era da IA

AI troubles are not only about information bias but also labour. (p. 20)

AI is not just a control apparatus, but also a productive one. (p. 20)

Trabalho fantasma

O trabalho no mercado de IA não é artificialmente autônomo, mas sim, humano e invisível.

Against the idea of alien intelligence at work, it must be stressed that in the whole computing process of AI the human worker has never left the loop, or put more accurately, has never left the assembly line. (p. 20)

Heteromação

Automation is a myth; because machines, including AI, constantly call for human help, some authors have suggested replacing ‘automation’ with the more accurate term heteromation. Heteromation means that the familiar narrative of AI as perpetuum mobile is possible only thanks to a reserve army of workers.

Ser-humano como recurso informacional da máquina

The information source of machine learning (whatever its name: input data, training data or just data) is always a representation of human skills, activities and behaviours, social production at large. (p. 19)

All training datasets are, implicitly, a diagram of the division of human labour that AI has to analyse and automate. (p. 19)

Datasets for image recognition, for instance, record the visual labour that drivers, guards, and supervisors usually perform during their tasks. (p. 19)

Even scientific datasets rely on scientific labour, experiment planning, laboratory organisation, and analytical observation. (p. 19)

IA como extração de inteligência analítica a partir de trabalho

The information flow of AI has to be understood as an apparatus designed to extract ‘analytical intelligence’ from the most diverse forms of labour and to transfer such intelligence into a machine (obviously including, within the definition of labour, extended forms of social, cultural and scientific production). (p. 19-20)

Divisão do trabalho -> automação do trabalho

In short, the origin of machine intelligence is the division of labour and its main purpose is the automation of labour. (p. 20)

O projeto de mecanização da divisão do trabalho

Historians of computation have already stressed the early steps of machine intelligence in the 19th century project of mechanizing the division of mental labour, specifically the task of hand calculation. The enterprise of computation has since then been a combination of surveillance and disciplining of labour, of optimal calculation of surplus-value, and planning of collective behaviours. (p. 20)

Computation was established by and still enforces a regime of visibility and intelligibility, not just of logical reasoning. The genealogy of AI as an apparatus of power is confirmed today by its widespread employment in technologies of identification and prediction, yet the core anomaly which always remains to be computed is the disorganisation of labour. (p. 20)

Regulação e proteção do trabalhador

The impact of AI on labour is well described (from the perspective of workers, finally) within a paper from the European Trade Union Institute, which highlights ‘seven essential dimensions that future regulation should address in order to protect workers: 1) safeguarding worker privacy and data protection; 2) addressing surveillance, tracking and monitoring; 3) making the purpose of AI algorithms transparent; 4) ensuring the exercise of the ‘right to explanation’ regarding decisions made by algorithms or machine learning models; 5) preserving the security and safety of workers in human-machine interactions; 6) boosting workers’ autonomy in human–machine interactions; 7) enabling workers to become AI literate.’ 71 (p. 20)

Educação pública sobre aprendizado de máquina

Análogo à questão de inserir a técnica na cultura em Simondon? [15]

Today an Intelligent Machinery Question is needed to develop more collective intelligence about ‘machine intelligence,’ more public education instead of ‘learning machines’ and their regime of knowledge extractivism (which reinforces old colonial routes, just by looking at the network map of crowdsourcing platforms today). (p. 20)

Objetivo do nooscópio

Por luz no trabalho invisível que faz com que inteligência maquínica pareça estar ideologicamente viva.

Also in the Global North, this colonial relationship between corporate AI and the production of knowledge as a common good has to be brought to the fore. The Nooscope’s purpose is to expose the hidden room of the corporate Mechanical Turk and to illuminate the invisible labour of knowledge that makes machine intelligence appear ideologically alive. (p. 20)

---

1. PASQUINELLI, Matteo; JOLER, Vladan. “The Nooscope Manifested: Artificial Intelligence as Instrument of Knowledge Extractivism”, KIM research group (Karlsruhe University of Arts and Design) and Share Lab (Novi Sad), 1 May 2020 (preprint forthcoming for AI and Society). https://nooscope.ai
2. CRAWFORD, K.; PAGLEN, T. Excavating AI: The politics of images in machine learning training sets. Excavating AI, 2019.
3. ZUBOFF, S. Big other: capitalismo de vigilância e perspectivas para uma civilização de informação. In: Tecnopolíticas de Vigilância. BRUNO, Fernanda; CARDOSO, Bruno; KANASHIRO, Marta; GUILHON, Luciana. [s.l: s.n.]. p. 17–68.
4. WALFORD, A. Raw data: Making relations matter. Social Analysis, v. 61, n. 2, p. 65–80, 2017.
5. LECUN, Y.; BENGIO, Y.; HINTON, G. Deep learning. Nature, v. 521, n. 7553, p. 436–444, maio 2015.
6. KOTIKALAPUDI, R.; CONTRIBUTORS. keras-vis. [s.l.] GitHub, 2017.
7. ALBER, M. et al. iNNvestigate Neural Networks! Journal of Machine Learning Research, v. 20, n. 93, p. 1–8, 2019.
8. TEOREMA DA APROXIMAÇÃO UNIVERSAL. In: WIKIPÉDIA, a enciclopédia livre. Flórida: Wikimedia Foundation, 2018. Disponível em: . Acesso em: 24 abr. 2018.
9. MIKOLOV, T. et al. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs], 6 set. 2013.
10. ROUVROY, A.; BERNS, T. Governamentalidade algorítmica e perspectivas de emancipação: o díspar como condição de individuação pela relação? Revista ECO-Pós, v. 18, n. 2, p. 36–56, 2015.
11. FOUCAULT, M. Aula de 17 de março de 1976. In: [s.l.] Martins Fontes, 2010.
12. FOUCAULT, M. Direito de morte e poder sobre a vida. In: História da sexualidade I. [s.l: s.n.]. p. 125–149.
13. DELEUZE, G. Post Scriptum sur les sociétés de contrôle. POURPARLERS, Éditions de Minuit, 1990.
14. GUATTARI, F.; ROLNIK, S. Cultura: um conceito reacionário. In: GUATTARI, Félix; ROLNIK, Suely. Micropolítica: cartografias do desejo. [s.l: s.n.]. v. 4p. 15–24.
15. SIMONDON, G.; FERREIRA, P. P.; KASPER, C. P. Introdução. CTeMe, [s.d.]. Disponível em: . Acesso em: 24 nov. 2020