Applying the Mingei protocol in the Glass pilot

STEP 1. Human resources and digital assets

In this step, we collect the documentation that will be eventually transformed into knowledge and history that we will digitally represent.

Human resources are invaluable in the description and explanation of craft practice and context. In this process, digital assets serve a range of purposes, from note-taking to reference practices by practitioners. Moreover, sometimes the contribution of a human resource may be available only through a recording.

Digital assets are carriers of documentation, information, and knowledge that required a comprehensive representation of craft. Moreover, digital assets that are appreciated by human observers (i.e., audio-visual) can encompass an insightful understanding of processes, environments and semiotics.

In this context, the following activities were conducted:

Knowledge collection and literature review

Craft understanding is initiated with the collection of existing knowledge of the following categories:

• Basic knowledge including (a) curated text, (b) curated material, and (c) formal representation
• Existing content includes (a) archives, (b) bibliographic research outcomes, (c) audiovisual material, and (d) digital representations
• Contextual information (1) brief description of the craft (2) geographical location and range of the craft, (3) selected and representative communities concerning the craft, (4) craft workers / skilled workers / handicrafts education, (5) gender roles, (6) equipment, (7) craft products, (8) craft traditions incuding oral tradition / story telling / work songs / myths & legends, (9) social practices / social dimension, (10) economic dimension, (11) design dimension, (l2) artistic dimension, (13) rituals, (14) festive events, (15) religious dimension, (16) learning / education / transmission of knowledge, (17) geography of workshops, (18) emic (inside) Presentation
• Narrative interviews with craft practitioners

Social and Historic Context

In 1842, Bontemps delivered more than fifty pieces, both tools (molds, and blowing apparatus) and finished objects. All pieces were dated from 1842 and originating from the Choisy-le-Roy glassworks. In 1868, Bontemps will make a further donation, in his name, objects relating to his famous Guide du Verrier, published that same year. Among them two sets of objects, revealing the process in a sort of «frozen state», a pattern much favored for in pedagogical dimension at the Musée des Arts et Métiers. The aim on selecting the carafe, in this work, was to document each manufacturing step, as exemplified by the «frozen» glass artifacts: the rough out foot and the finished one, the handle, etc. with a recording of the corresponding gestures and bring to life the objects. Furthermore, it was interesting to compare and document the historical tools with those still in use today.

Data collection

In this step, the components of the crafting process are digitally recorded, post-processed, and curated in an online platform.  

These digital assets record objects and actions, acquired from documenting photographically and in 3D materials, tools, products, and workspaces and recording practitioner crafting actions, in recording sessions. The organization of these sessions is facilitated by the vocabulary and storyboard, in identifying the objects, sites, and practitioner actions to be digitized. Moreover, some of the photographic and video assets may be recorded in combination with ethnographic fieldwork. A requirement relevant to the glasswork, but also relevant to other crafts, is that the viscous nature of hot glass prohibits the pause of the crafting process during some sequences of tasks. As such, motion files may contain multiple process steps and actions

Audiovisual recordings

Pertinent assets regard conventional audiovisual data acquired from the ethnography such as audio and video interviews photographic documentation, as well as documentation to be used for craft representation including photographs and video documentation of objects, spaces, and demonstrations.

Audiovisual recordings of the crafting process are important for overview and ethnographic study, as well as for documenting significant aspects of the process with a local spatial expression, such as tool gripping postures, detailed and minute manipulations, and so on.

Photographic documentation

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Video documentation

Multiple and/or wide-angle bird view cameras were employed to capture the motion of multiple practitioners and their movement within the workshop. In the case of glass workshops, the latter is particularly important because glass workshops are shared by multiple practitioners and because the arrangement of workshop machinery is such that the practitioners have to walk from one locus to another during the crafting process.

Documentation of tools, machinery, and workspaces

Tools, machinery, and workspaces was documented using photographic documentation.

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Motion capture

Motion Capture occurred during the ethnographic fieldwork at CERFAV, in September and October 2019. During this, fourteen carafes were produced in the workshop to observe and capture social and body interactions that take place during glasswork. Apart from the audiovisual documentation acquired the process was recorded using MoCap equipment and, in particular, the Nansense MoCap suit and Glove. The following gallery illustrates some of the tasks that were recorded for glassblowing.

• 

• 

• 

• 

• 

• 

• 

• 

Post processing of Digital Assets

Ethnographic documentation

Video analysis and identification of key operational sequences that signify craft actions and processes. Definition of a process flow diagram for the implementation of the glass carafe and connection of operational sequences with the flow diagram. Examples of the ethnographic documentation of such sequences are presented in the following gallery.

Visual tracking results

For visual tracking, the OpenPose library and the MocapNET body tracking software was used. On the Glass datasets we observe that, when all limbs are visible, motions tend to translate well into the armature. Failure frames mostly consist of several limbs being occluded and not visible in the camera observations. Invisible joints lead to missing 2D joint estimations and missing 2D estimations, in turn, mean empty lines and columns in the NSDM matrices of MocapNET and not enough structural information for a clearly defined solution.

Examples of the visual tracking outcomes are presented in the following gallery:

MoCap segmentation

Digitisation Outcomes

New digital assets

Glassblowing tools

The 3D documentation of metallic tools and glassware can be challenging. We distinguish between the potential historic significance of an object and its utility as a tool, wherein the latter solely their geometrical structure can be sufficient. In some cases, the 3D model of tools can be easily modeled or even provided by the manufacturer. The use of synthetic models can simplify the digitization tasks and significantly reduce scanning costs. The collection of tools for glasswork was developed in and is summarized in the following gallery.

Workshop equipment

Similarly, for the creation of 3D models of machinery, the acquired audiovisual documentation acquired during the ethnographic research was used and the machines were created from scratch using 3D modeling software.

Relevant Publications

• Representation and Preservation of Heritage Crafts, (2020), X. Zabulis, C. Meghini, N. Partarakis, C. Beisswenger, A. Dubois, M. Fasoula, V. Nitti, S. Ntoa, I. Adami, A. Chatziantoniou, V. Bartalesi, D. Metilli, N. Stivaktakis, N. Patsiouras, P. Doulgeraki, E. Karuzaki, E. Stefanidi, A. Qammaz, D. Kaplanidi, I. Neumann-Janßen, U. Denter, H. Hauser, A. Petraki, I. Stivaktakis, E. Mantinaki, A. Rigaki, G. Galanakis, Sustainability, DOI:10.3390/su12041461.
• What is needed to digitise knowledge on Heritage Crafts?, (2019), Zabulis, X., Meghini, C., Partarakis, N., D. Caplanidi, P. Doulgeraki, Karuzaki, E., Evropi Stefanidi, Evdemon, T., Metilli, D., Bartalesi, V., Fasoula, Maria, Tasiopoulou, E., Beisswenger, C., Memoriamedia 1: 1–25.

Applying the Mingei protocol in the Glass pilot

STEP 2. Knowledge Elements

In this step, basic craft knowledge or otherwise knowledge elements are formed and digitally represented. In plain words, this means that the digitisations are entered in the Mingei repository, properly classified and linked to the other entities in the repository. Collectively, the classifications and links formed in this step form the semantic metadata that have been previously introduced.

Forming a knowledge element requires a comprehensive understanding of the digitally represented asset and is regarded as a digital curation process. Digital assets are represented in the system by knowledge elements associated with the result of a curation process that yields metadata, annotations, and descriptions. In this process, the original meta-data (if any) that accompany the digital asset are useful.

Representation of knowledge on craft processes includes curated material which is also digitally represented. The basic elements of such representations are the tools, machines, workshops, etc. At the same time of importance is the social and historical context.

Social and Historical Context

Craft Context

Digital assets

The most relevant digital data types are:

• Text
• Photographs
• Video
• Audio
• 3D reconstruction
• Human motion digitisation

Additional digitisation types of humans, matter, and actions (e.g. thermal, multispectral, X-ray), provide measurements of more material properties that photography can capture. Such properties are surface reflection and transmission properties, chemical composition, its degree of viscosity at a certain temperature, and other.

We provide some examples of primary digital assets from the Mingei dataset

• Digital text of curated text, literature, and testimonies.
• Photographs: historic photographs, photographic documentation of artefacts, printed matter, records.
• Audio: interviews of glassmaker and mastic producer.
• Video of craft practice.
• 3D reconstruction of a workshop from a laser scan.
• 3D reconstruction of an artefact from a handheld scanner.
• Motion capture of craft practice.

Relevant Publications

• Representation and Preservation of Heritage Crafts, (2020), X. Zabulis, C. Meghini, N. Partarakis, C. Beisswenger, A. Dubois, M. Fasoula, V. Nitti, S. Ntoa, I. Adami, A. Chatziantoniou, V. Bartalesi, D. Metilli, N. Stivaktakis, N. Patsiouras, P. Doulgeraki, E. Karuzaki, E. Stefanidi, A. Qammaz, D. Kaplanidi, I. Neumann-Janßen, U. Denter, H. Hauser, A. Petraki, I. Stivaktakis, E. Mantinaki, A. Rigaki, G. Galanakis, Sustainability, DOI:10.3390/su12041461.
• What is needed to digitise knowledge on Heritage Crafts?, (2019), Zabulis, X., Meghini, C., Partarakis, N., D. Caplanidi, P. Doulgeraki, Karuzaki, E., Evropi Stefanidi, Evdemon, T., Metilli, D., Bartalesi, V., Fasoula, Maria, Tasiopoulou, E., Beisswenger, C., Memoriamedia 1: 1–25.
• An Approach to the Creation and Presentation of Reference Gesture Datasets, for the Preservation of Traditional Crafts, (2020), N. Partarakis, X. Zabulis, A. Chatziantoniou, N. Patsiouras, I. Adami, Applied Sciences, DOI:10.3390/app10207325.

Open Datasets

• Carlo Meghini, Valentina Bartalesi, Daniele Metilli, Nikolaos Partarakis, & Xenophon Zabulis. (2020). Mingei Ontology (1.0). Zenodo. https://doi.org/10.5281/zenodo.3742829
• Nikolaos Kokolantonakis. (2020). Requirements and protocol of the 3D documentation of the Heritage Crafts tools [Zenodo]. https://doi.org/10.5281/zenodo.3813287

Applying the Mingei protocol in the Glass pilot

STEP 3. Craft Representation

In this step, the individual entities represented in the previous steps are linked to each other into an organic representation of the craft instance. This linkage implements the semantic representation of the craft.

The scope of this representation covers the following craft dimensions:

• Tangible elements, such as materials, tools, and products
• Craft actions and processes
• Contextual knowledge that provides an understanding of
  • Artefact usage
  • The CH of a region and its people embedded in the artefact
  • The historic, geographical, economic, and social dimensions of the associated craft instance.

Social and Historic context

Craft Representation

To encode craft understanding, activity diagrams are borrowed from Unified Modeling Language (UML) and used in the following sense. While UML represents computational actions that transform data are represented, in this work, physical actions that transform materials are. The transition types Transition, Fork, Merge, Join, and Branch are adopted and denoted as in UML.
Activity diagrams can be refined hierarchically, allowing the increase of representation detail at later stages. Moreover, their visual nature was found to support the collaboration with practitioners. Activity diagrams are essential in representing transitions between process steps, in which distinct tools are employed and actions take place. and, in particular, decision points and parallel tasks in the process.
The progression of sequential steps is modeled by a Transition link. Forks are used to represent the initiation of two parallel tasks. In Merge transition, two or more control paths unite and Join connects steps that should be completed before the transitions to the next step. Merge and Join transitions are structurally similar, but a join is a synchronization across a set of parallel flows, while in a merge only a single flow is active. Finally, Branch transitions connect a step with a decision step that accepts tokens on one incoming edge and selects one outgoing alternative. Branch nodes control the flow of a process by selecting one of several alternatives, based on the outcome of a condition evaluation.

Relevant Publications

• Digitisation of traditional craft processes, (in press, JOCCH-21-0060), X. Zabulis, C. Meghini, A. Dubois, P. Doulgeraki, N. Partarakis, I. Adami, E. Karuzaki, A. Carre, N. Patsiouras, D. Kaplanidi, D. Metilli, V. Bartalesi, C. Ringas, E. Tasiopoulou, Z. Stefanidi, ACM Journal on Computing and Cultural Heritage, DOI:10.1145/3494675.

Open Datasets

• Carlo Meghini, Valentina Bartalesi, Daniele Metilli, Nikolaos Partarakis, & Xenophon Zabulis. (2020). Mingei Ontology (1.0). Zenodo. https://doi.org/10.5281/zenodo.3742829

Applying the Mingei protocol in the Glass pilot

STEP 4. Narratives

A digital narrative or, simply, a narrative is an abstraction that represents a set of facts that have happened in the world. These facts are connected to each other in a way that makes them a story.

This abstraction includes a narration, which is the way that a certain author has told the story. Individual narratives may be focused at a subject and include a pertinent subset of the events that are represented in the fabula.

The narration tells the story, using a certain medium. In Mingei multiple media are employed, starting from verbal and visual and reaching up to immersive and interactive narrations. A medium, and correspondingly, a media object may be multidimensional; for example the audio and visual components of a mobile phone “video” recording, or the combination of video and 3D MoCap recordings.

A narrator is a person or a program that presents a media object, i.e., verbally, by visual rendering, sonification, VR, etc. Besides multimedia and immersive presentations (discussed in the next step), conventional, verbal and visual, media objects and channels are of fundamental importance. A textual media object can be exported as a narrative for a person or an avatar to read, and a formatted document with illustrations, or an “infographic”, can output channels. In this context, Mingei aspires the export of more imaginative and interactive uses of conventional verbal and visual outputs, including scripts for theatrical re-enactment of stories and role-play on contextual dimensions of craft instance, such as skill, trade, and labour. The choice of narration medium is relevant to the use of the content, the intended audiences, and the experiences to be rendered these are discussed in the next step. In this step, we focus on the narrative representation.

A narrative can include additional information besides the connection of the narration to the facts that are narrated. In particular, a digital narrative consists of three main elements:

1. A representation of the fabula made of representations of the events that the narration is telling.
2. One or more media objects.
3. A digital representation of the reference function that connects the (representations of the) events of a story with a fragment of a media object that describes them and allows to derive the plot.

Each narration of a fabula employs the use of one or more media objects, which have been authored by one or more narrator(s). Through the Mingei Online Platform, a tool for authoring and visualising narratives is provided.

Representation of Narratives

Relevant Publications

• Representation of socio-historical context to support the authoring and presentation of multimodal narratives: The Mingei Online Platform, (2021), N. Partarakis, P. Doulgeraki, E. Karuzaki, I. Adami, S. Ntoa, D. Metilli, V. Bartalesi, C. Meghini, Y. Marketakis, M. Theodoridou, D. Kaplanidi, X. Zabulis, ACM Journal on Computing and Cultural Heritage, DOI:10.1145/3465556.
• Meghini, C., Bartalesi, V., & Metilli, D. Representing Narratives in Digital Libraries: The Narrative Ontology.
• Representing narratives in digital libraries: The narrative ontology, (2021), C. Meghini, V. Bartalesi, D. Metilli, Semantic Web, DOI:10.3233/sw-200421.
• Representation and Presentation of Culinary Tradition as Cultural Heritage, (2021), N. Partarakis, D. Kaplanidi, P. Doulgeraki, E. Karuzaki, A. Petraki, D. Metilli, V. Bartalesi, I. Adami, C. Meghini, X. Zabulis, Heritage, DOI:3390/heritage4020036.

Applying the Mingei protocol in the Glass pilot

Step 5. Information tools

Implementation of Virtual Humans

For the needs of the demonstrations 3D virtual humans, with their corresponding garments and accessories, have been carried out based on cultural and historical information and sources. VHs were created using references from the ethnographic research. For the definition of the 3D meshes and the design of the skin surfaces a customized toolchain was employed that combines automatic generation methods, with manual editing and refining. The approach utilizes VH creation software and 3D modeling software for clothes and accessories. The final model inherits and encapsulates the required structural components, such as bone and skin attachment data. This allows the VH to be dynamically animated and to exhibit real-time animation capabilities.

Another important aspect for attaining a high level of realism is the motion, behavior, and natural interaction of the VH with users, to avoid the “uncanny valley” effect. The VH must have the ability for verbal as well as nonverbal communication skills, be intelligent, have natural communication with the users, perceive information from the user, and physically react with them. The deformation of the skinned characters during their movements must be realistic, smooth, and not contain any discontinuities. The VH that supports all these capabilities will be an intuitive craft master acting effectively as an elderly companion and assistant for elder people.

Garment design was also considered during the implementation of avatars. To ensure appropriate rigging, since the generated models are automatically rigged an additional checking was performed to ensure that the rig is applied correctly, and the bones are well adjusted to the 3D body.

Motion Re-targeting

The processing of the motion data has been done on Autodesk MotionBuilder software which is dedicated to animation and the direct integration of motion capture technologies. The process requires different steps: (a) creation of an “actor” in MotionBuilder with skeleton definition corresponding to the BVH hierarchy, (b) mapping of the received animations on the actor, and (c) synchronization of the VH with the actor by adjusting the 2 models so that the measurements match and the animations are correctly reproduced. Examples of the outcomes are presented below:

Web based information tools

The rich semantic representation of the craft is transformed to intuitive online presentations and delivered through the Mingei Online Platform.

Illustrated craft demonstrations

Relevant Publications

• Transferring Traditional Crafts from the Physical to the Virtual World: An Authoring and Visualization Method and Platform (in press, JOCCH-20-0165), E. Stefanidi, N. Partarakis, X. Zabulis, I. Adami, S. Ntoa, G. Papagiannakis, ACM Journal on Computing and Cultural Heritage, DOI:10.1145/3484397.
• Realistic Virtual Humans for Cultural Heritage Applications, (2021), Karuzaki E, Partarakis N, Patsiouras N, Zidianakis E, Katzourakis A, Pattakos A, Kaplanidi D, Baka E, Cadi N, Magnenat-Thalmann N, Ringas C, Tasiopoulou E, Zabulis X., Heritage, 4(4):4148-4171. DOI:10.3390/heritage4040228.
• An Approach Towards Artistic Visualizations of Human Motion in Static Media Inspired by the Visual Arts, (2020), A. Rigaki, N. Partarakis, Zabulis, C. Stephanidis, International Conference on Advances in Computer-Human Interactions (ACHI).
• An Approach for the Visualization of Crafts and Machine Usage in Virtual Environments, (2020), E. Stefanidi, N. Partarakis, X. Zabulis, G. Papagiannakis, International Conference on Advances in Computer-Human Interactions (ACHI).
• TooltY: An Approach for the Combination of Motion Capture and 3D Reconstruction to Present Tool Usage in 3D Environments, (2020), E. Stefanidi, N. Partarakis, X. Zabulis, P. Zikas, G. Papagiannakis, N. Magnenat-Thalmann, Intelligent Scene Modeling and Human-Computer Interaction, DOI:1007/978-3-030-71002-6_10.

Open Datasets

• Anastasia Rigaki, Nikolaos Partarakis, & Xenophon Zabulis. (2021, May 12). The making of the Bontemps carafe. Zenodo. https://doi.org/10.5281/zenodo.4750880
• Anastasia Rigaki, Nikolaos Partarakis, & Xenophon Zabulis. (2021, May 11). The making of the Bontemps’ carafe. Zenodo. https://doi.org/10.5281/zenodo.4749179

Applying the Mingei protocol in the Glass pilot

Step 6. Experiences

Mixed Reality

The Mixed reality installation was deployed in the context of a periodic exhibition at the premises of the museum of CNAM, which is is where the worlds of academics and professional activity come together. It is the only higher education establishment dedicated to life-long professional training. In this context, a dedicated space at the cathedral which is part of the museum was dedicated for the exhibition. The space is a chapel of the cathedral located within the main exhibition of the museum. The installation is comprised of a special construction capable of hosting three large displays created through back projection in thin synthetic fabric. Furthermore, for craft training, a bench has been installed in front of the main display together with a glassblowing pipe to be used by visitors. The installation has integrated hosting spots for the glass carafes and pieces of the carafe created during the ethnographic fieldwork. Special lighting was integrated within to present the interaction of glass with light and to create atmosphere.

After the completion of the physical part of the installation, the software was installed. The scenario that was set up was that the presentation application sequence is executed first and then the main projection switches to the craft training applications, so that users test what they learned from the craft presentations.

• 

• 

• 

• 

• 

• 

• 

• 

Mobile AR

Relevant Publications

• Mixed-Reality Demonstration and Training of Glassblowing, (2022) Carre, A. Dubois, N. Partarakis, X. Zabulis, N. Patsiouras, E. Mantinaki, E. Zidianakis, N. Cadi, E. Baka, N. Thalmann, D. Makrygiannis, A. Glushkova, S. Manitsaris, , Heritage 2022, 5(1), 103-128, doi:10.3390/heritage5010006.
• Stochastic-Biomechanic Modeling and Recognition of Human Movement Primitives, in Industry, Using Wearables, (2021), B. Olivas-Padilla, Sotiris Manitsaris, D. Menychtas, A. Glushkova, Sensors, DOI:3390/s21072497.
• Human Movement Representation on Multivariate Time Series for Recognition of Professional Gestures and Forecasting Their Trajectories, (2020), Manitsaris, Gavriela Senteri, D. Makrygiannis, A. Glushkova, Frontiers in Robotics AI, DOI:10.3389/frobt.2020.00080.
• Extracting the Inertia Properties of the Human Upper Body Using Computer Vision, (2019), D. Menychtas, A. Glushkova, S. Manitsaris, Computer Vision Systems (CVS), Thessaloniki, Greece, September 23–25, DOI:10.1007/978-3-030-34995-0_54.
• Transforming Heritage Crafts to Engaging Digital Experiences, (2020), N. Partarakis, X. Zabulis, M. Antona, C. Stephanidis, Visual Computing for Cultural Heritage, DOI:1007/978-3-030-37191-3_13.