In museums, objects are often exhibited separately. Tools are shown alongside other tools, and glass objects are exhibited together with other glass objects. How do you tell a coherent and captivating story, connecting the dots between different exhibited objects? In co-creation with CNAM (Conservatoire national des arts et métiers), Waag has been prototyping a digital experience for Mingei’s pilot on glass. Developer Lodewijk Loos takes you along the journey towards the first prototype.

Visiting CNAM

The goal of creating this digital experience at CNAM is to engage visitors and give them insight into the process of glass making. The prototype should work on site (in this case in the context of the museum), and should add to the already available real objects on display. However, the technology used for the prototype should be non-obtrusive to the local situation. Visitors who do not wish to use the technology, should not be bothered by it.

In order to get a grasp of the local context at CNAM, Meia Wippoo and Lodewijk Loos of Waag went to Paris in March 2020. There, we had a fruitful co-creation session with a team of museum professionals from CNAM and conceptualized a rough version of the prototype. In the glass section of the museum, we made some observations that were key to the first version of the prototype.

First of all, the glass objects are exposed in vitrines, they couldn’t be touched or picked up and could not be looked at from all angles. Of course, not being able to pick up objects in a museum is normal. However, as a lot of these objects are tools and utensils, being able to do so would contribute to the understanding of the object. We also know from experience in earlier projects, like meSch, that being able to pick up objects leads to more user engagement.

The next thing we noticed is that some objects were related to other objects that were displayed in different rooms of the museum. For example, the glass tools and a glass product were not in the same room. The reason for this is that there are different ways to classify object. The tools were in the tooling section and the carafe was in a section with artworks. However, these objects are part of the same story that we would like to tell: the process of glass making.

Another observation that we made was that some of the objects key to the story were not on display in the museum, for example a furnace and piece of wet paper were not there.

Augmented reality

As we decided upfront, the prototype should help to get insight in the process of glass making. With these observations, we could translate the story of glass in a more generic story. One could say that in the context of crafts, a general pattern is that objects are used with other objects (for example tools with materials), in different parts of the process. That is what we want our digital experience to give insight in.

We also concluded that the use of augmented reality (AR) technology could be of value for this prototype. With AR, it is possible to create the sense of picking up (virtual representatives of) objects, use them in another room, and show objects that are not physically there.

Mark the process

We returned back home and worked out several concepts. Next, we aggregated common interaction principles from our concepts. Our interaction principles showed similarities to (adventure) games. Adventure games are like a puzzle: you often have to pick up objects, sometimes not yet knowing what for, and use them at another location, sometimes in combination with another object.

One of our concepts focused at the carafe, named “Mark the Process”. This concept would lend itself for this type of adventure-like (mini) game. The central piece in this game would be the various parts and stages of completion of the carafe. This is how the process of making this type of carafe is currently displayed in CNAM. Wouldn’t it be nice if you had pick up the tools associated with this process in the one room, and place them at the right “step” in the other room? We also liked the idea of being able to collect museum objects and take them home for closer inspection.

The use of markers

With this concept in mind, we started implementing a proof of concept as a smartphone app. From previous AR projects, we had experience with the combination of Vuforia (AR framework) and Unity3D (gaming engine). The former is very well integrated in the latter, making it an ideal tool for (at least) prototyping. Vuforia support various ways of augmentation, both marker-based as marker less.

Markers are physical signs that are recognized by the app to instigate interaction. Using markers makes the app less dependent on local lighting conditions, which were not ideal or constant at CNAM. Recognising a marker, instead of an object itself, generally just works better. Additionally, using markers could make it easier for users of our app to see at which locations in the museum they could interact, because they serve as a visual clue. When you’re in a museum with thousands of objects, it is convenient that you can see immediately (without using a device) which ones are interactable. Finally, markers are easier in use. Augmenting an object by placing a marker in front of it is less challenging then having to scan the object and markers make it also easy to place objects in the void. In the longer run, the use of markers helps to accomplish a more generic application for different venues with different content, that allows its content to be authored by curators (as opposed to software developers).

Living room demonstration

Our original intention was to test the prototype at CNAM with random visitors of the museum. But during the development of the app, Covid-19 came around and it became clear that testing the app in a public venue with a real audience would not be possible anytime soon. Furthermore, the Covid-19 situation might even change the way we design things permanently. For example, it might have become undesirable to have devices in a museum that are handed out to visitors or to have installations with touch screens. An AR app that people can run on their own phone should be relative safe and convenient.

With this in mind, we slightly changed our prototyping strategy and made the decision to create a living room demonstration. Originally, the prototype was meant to include virtual copies of the museum objects. By the lack of museum objects in the developer’s house, we used general building tools and convincing 3D models from online repositories.

The prototype demonstrates a few of the principles. The user can pick up object and place them back again, objects can be collected in a treasure chest for later use, objects can be used with other objects by using them with a marker next to that other object, referenced media for the collected object is available as background information, information overlays (giving hints) can be shown and a collection of objects can be used to make simple puzzles. As a gamification element, the user receives badges after completing specific tasks or reaching certain goals.

Next steps

This simple approach allows for a lot flexibility to create puzzle-like games. For example, a timeline game could be created by changing the physical placement of the markers into another linear layout. One could also imagine having different kinds of visual markers for different kind of interactions. One type of marker could indicate that an object can be picked up, and another marker could indicate that an object can be used at that spot.

At this point it is also interesting to think about how these principles can be applied at the other pilot locations. Part of the Mingei project is a pilot in Chios (Greece) on the craft of harvesting and processing Mastic from the mastic tree. Would it be feasible to apply the prototype at the local situation over there by augmenting the Mastic tools and placing markers on and around a real tree? There is still enough work to be done and questions to be answered towards a generic AR application for on-site craft experiences!

Written by Lodewijk Loos (Waag)

As part of the Mingei project, the Musée des Arts et Métiers is making a copy of a carafe dated in 1842, made by Georges Bontemps at his crystal factory of Choisy-le-Roi in France. This work is documented by an anthropologist of the Laboratory “Histoire des Techno-Sciences en Société” of CNAM. The recording of the glass master’s gestures using a sensor suit and different cameras is done by ARMINES Research Center in Paris and the Institute of Computer Science FORTH in Heraklion, Greece.

During the month of May, the Centre Européen de Recherches et de Formation aux Arts Verriers (CERFAV) already studied the re-enactement process of the carafe. In September and October, the teams worked together to reproduce the carafe and record the actions necessary to craft the 1842 Bontemps’ carafe.

The manufacturing of the carafe

The master glassmaker Jean-Pierre Mateus and Dominique Jamis, former head of the hot-glass workshop at CERFAV, experimented with several techniques to assimilate the old carafe manufacturing process. More than twenty carafes were made before reaching a model similar to the original one. The manufacturing steps were as follows:

1. Blowing of the body: first step of the manufacturing pr
   ocess, the glassblower picks the glass in the furnace and shape it by his breath, the gravity and his hands to create the desired body of the carafe.
2. Laying the ” leg and foot ” of the carafe: with the help of the assistant, this step involves sticking two new glass parts to the body of the carafe by returning it and deposing two very specific amount of glass to be able to shape it with tools.
3. Crafting the neck: with a pincer, the glassmaster stretches the upper part of the body to made it identical to the original model.
4. Laying the glass cord: the assistant comes to deposit a fine tub between the body and the neck and the glassmaster shapes it while the glass is still hot.
5. Cutting the beak of the carafe: with a chisel, the glassmaster opens the neck of the carafe to desired shape and forms a beak.
6. Laying the handle: the assistant brings hot glass and puts it on the beak of the carafe, then he stretches it to create a tube that the glassmaster attaches to the body with the desired shape.

Two tools had to be specially created during the re-enactment process. First, a clapper in wood consisting of two rectangular pieces of wood joined at one end by a leather hinge and an aperture in one of the pieces of wood to squeeze a blob of glass in order to form the foot. Second, an experimental tool in soft metal created by the glassmaster to be able to make the cord between the body and the neck as thin as it is in the original carafe.

Ethnographic observation

During this entire process, the anthropologist Arnaud Dubois has conducted an ethnography of this re-enactment consisting of an observation of the working activities, the documentation of these activities with photographs and films, and different formal and informal interviews of the two craftsmen. One of the methodologies used during this research interest has been to observe and document the “geography of the workshop” to understand the relation between the body, the tools, the matter and the space when the craft is performed. Using the general frameworks of the “operational sequence” as conceptualised by the French anthropologist André Leroi-Gourhan in 1965, this method helps the researcher to understand the complex interaction between the craft practitioners gestural and sensitive actions, the active role of matter and tools, and the ‘choreographic’ dimension of movement in the workshop as a fundamental characteristic of a craft practice. Because craft knowledge is mostly non-verbal, this methodology is very useful to define and document the technical gesture of a craft.

While the operational sequence has been understood by the anthropologist, members of ARMINES laboratory equipped the glassmakers with sensors to record their key gestures and monitor their breath. Then, five cameras, installed in the workshop by the engineers of FORTH, were used to document the interaction between the master glassmakers and complete the recording of their actions.

Multidisciplinary collaboration

The collaboration between craftsmen, research-engineers and a social scientist permits to understand, reproduce, describe and document the tangible and intangible know-how embedded in the heritage artefact of the CNAM’s collection. The multiple discussions and several experimentations between these people from different background and disciplines create a deep and new understanding of the carafe. An interesting thing of this collaboration has been also to help everyone to have a reflexive approach of his own work and methodology. The craftsman through his collaboration with the computer scientists rethought his technical gesture in a new way. The anthropologist who is working with the engineers thinks differently about the ethnographic methodology he could use to study technical gestures. And the research-engineers collaborating with an ethnographer need to be more precise and exhaustive in their digitisation.

In November, Arnaud Dubois will go to FORTH to work with the technological partners on the various materials collected. This collaboration between researchers in computer science and anthropology researchers aims to achieve a most accurate and realistic digital reproduction.

Written by CNAM