Knowledge Organisation


Kim H. Veltman

The Universe and the Multiverse: From Matching to Cross-Matching,”

Written as an Opening Keynote: Human-Computer Interaction, Tourism and Cultural Heritage (HCITOCH 2017). Bologna, 6-8 September, 2017: Then the organizer lacked funds to pay the flight.


There are different goals of art and image making. This paper considers seven, but focusses on two. The pre-historical and ancient world knew three: connecting, ordering and imitation (mimesis). The Renaissance developed a new goal of matching: copying and representing the universe, which became linked with perspective, truth and objective science. The 19th century introduced two further goals; mixing and exploring. Since 1996, there has been a seventh goal: cross-matching: one copies items in the universe to cross-match them with objects in a simulated world or metaverse.

A superficial consequence of this new goal is a surge in special effects in cinema: e.g. the bullet dodging protagonist of the Matrix (1999) or Jake Sully in Avatar (2009). At a deeper level, the rivalry between matching and cross-matching leads to a rivalry between reality and simulation; between matching of the universe and cross-matching with the metaverse; between reality/truth and illusion. A second consequence, has been an enormous surge in publications on perspective. One might expect a similar surge in methods to distinguish the two worlds. Instead, protagonists in social networks and social media are actively attempting to confuse, blur and conflate distinctions between these worlds. The paper explores possible consequences for future interfaces. Most discussions are in terms of new possibilities for communication, entertainment, gaming as if this were merely one further step towards an Internet of Things (IOT). Our concern is different: to warn that these developments threaten our concepts of truth and indeed endanger fundamental premises of what it means to be human. We need new forms of reality and truth meters.


Every society has approaches to image making. This paper considers seven (figure 1-2). In rare cases, they are iconoclastic and forbid image making altogether. Traditionally, early societies tend to follow a goal of connecting: whereby the image is less a copy of the original and serves to connect image and original via the trance of a shaman, or sympathetic and/or other forms of magic. The Greco-Roman world brought two other goals into focus: ordering and imitation (mimesis). The Renaissance introduced a fourth goal, which many have assumed was merely a rebirth of the earlier imitation, but was not.  Mimesis imitated a series of features in individuals with no commitment to copying the features of any given individual. Matching was committed to copying the features of a single individual person or object. This was much more than a simple reproduction process of a given person/object. In establishing a fixed relation between person/object, picture plane and observer, perspective led to an objective relation between individual and the world. The good news was that it established an objective relation with the world that proved vital for the rise of early modern science. The less good news was that the role of individual observer was reduced to an insignificant, passive vanishing point.

The 19th and early 20th centuries introduced two new goals. One was mixing, whereby artists played with the traditional laws of transparency and opacity. The other was exploring, which included the mental world, the perceptual world, and non-realistic art. In 1996, there was an unexpected development. Paul Debevec, invented a new method of reflection mapping. On the surface, it began simply as a clever method, whereby one could use the reversibility principle of perspective in photographs in order to reconstruct the original space. However, the same technology could be adapted such that instead of creating a new matching with the physical world, one could create a cross-matching with a simulated world.

This breakthrough has seen a surge in special effects in cinema, some of which will be noted. But it is the philosophical implications that are of interest. Cross-matching is not merely another goal. It competes with and potentially threatens to replace matching. In theoretical terms, this would mean replacing the quest to copy and replicate the physical world (universe) with creating a simulated world (metaverse). Since we live as physical beings in a physical world this is likely to remain theory to some extent but its potential implications are nonetheless profound.

Of concern, is that the champions of social media, notably, Facebook, are determined to confuse and conflate the physical and simulated worlds to the extent that we are no longer able to determine which is which. There is a quest to interface directly with our thoughts. The rhetoric is a quest for direct brain to brain communication.i In practice, this means that those who control the technology can listen in and spy even on our thoughts and dreams. As some have already noted, this poses dangers of direct AI propaganda.


Read full article