Mastic – Implementation timeline

Implementation timeline

 
 
 
 
 

Applying the Mingei protocol in the Mastic pilot

STEP 1. Human resources and digital assets

In this step, we collect the documentation that will be eventually transformed into knowledge and history that we will digitally represent.

Human resources are invaluable in the description and explanation of craft practice and context. In this process, digital assets serve a range of purposes, from note-taking to reference practices by practitioners. Moreover, sometimes the contribution of a human resource may be available only through a recording.

Digital assets are carriers of documentation, information, and knowledge that required a comprehensive representation of craft. Moreover, digital assets that are appreciated by human observers (i.e., audio-visual) can encompass an insightful understanding of processes, environments and semiotics.

In this context, the following activities were conducted:

Knowledge collection and literature review

Craft understanding is initiated with the collection of existing knowledge of the following categories:

  • Basic knowledge including (a) curated text, (b) curated material, and (c) formal representation
  • Existing content includes (a) archives, (b) bibliographic research outcomes, (c) audiovisual material, and (d) digital representations
  • Contextual information (1) brief description of the craft (2) geographical location and range of the craft, (3) selected and representative communities concerning the craft, (4) craft workers / skilled workers / handicrafts education, (5) gender roles, (6) equipment, (7) craft products, (8) craft traditions incuding oral tradition / story telling / work songs / myths & legends, (9) social practices / social dimension, (10) economic dimension, (11) design dimension, (l2) artistic dimension, (13) rituals, (14) festive events, (15) religious dimension, (16) learning / education / transmission of knowledge, (17) geography of workshops, (18) emic (inside) Presentation
  • Narrative interviews with craft practitioners

Social and Historic Context

PIOP holds the archive of the Chios Gum Mastic Growers Association who is the main stakeholder regarding mastic cultivation in Chios, along with the mastic growers themselves. Through this archive and PIOP’s library, literature, audio-visual material, as well as digitised objects such as 2D scans of historical books were retrieved. For the collection of new assets audio-visual material was produced, as well as 3D scans of the museum’s items, the mastic tree, and open spaces such as the Chios Mastic Museum and mastic villages. Furthermore, motion capture of the cultivators performing agricultural tasks took place. Fieldwork was also conducted at the mastic villages in order to collect more information regarding the significance and the perspective of the mastic growers in relation to mastic and its cultivation.

We have identified the following categories and thematics stemming from archival and on-site research in Chios:

  • Description of the cultivation craft and the processes performed by the growers and the Chios Gum Mastic Growers Association.
  • A dictionary of the equipment that is used in the different processes by the different actors, and the products that are produced with mastic.
  • We identify the distinctive geographical places in Chios that relate to mastic cultivation.
  • Socio-historically, we identified categories that strongly relate to the economy, history, and social organization of the island.
  • Festive events, oral traditions and customs, gender roles in craft processes, local religious beliefs, artistic expression related to Pyrgi’s (mastic village) decorative architecture, traditional clothing, as well as media representations (film, photography, press) of life in the mastic villages were discovered. Furthermore, emic (inside) presentations were collected through both archival research and on-site fieldwork. 

Data collection

In this step, the components of the crafting process are digitally recorded, post-processed, and curated in an online platform.  

These digital assets record objects and actions, acquired from documenting photographically and in 3D materials, tools, products, and workspaces and recording practitioner crafting actions, in recording sessions. The organization of these sessions is facilitated by the vocabulary and storyboard, in identifying the objects, sites, and practitioner actions to be digitized. Moreover, some of the photographic and video assets may be recorded in combination with ethnographic fieldwork.

Audiovisual recordings

Pertinent assets regard conventional audiovisual data acquired from the ethnography such as audio and video interviews photographic documentation, as well as documentation to be used for craft representation including photographs and video documentation of objects, spaces, and demonstrations.

Audiovisual recordings of the crafting process are important for overview and ethnographic study, as well as for documenting significant aspects of the process with a local spatial expression, such as tool gripping postures, detailed and minute manipulations, and so on.

Photographic documentation

Mastic museum

Context – Mastic Villages

Architecture
Pyrgi
Olympoi
Mesta
Elata

Mastic Houses

Architecture
Inside a mastic house

Results of surface scanning are available in the following video demonstrations:

Video documentation

For acquiring video documentation, in terms of equipment, we used three cameras for data acquisition. One digital camera (Nikon J1) with a 10-30mm lens that captured single images of resolution 3872 x 2592 as well as full HD (1920 x 1080) video at 60fps and two GoPro cameras with an ultra-wide lens capable of 2.7K (2704 x 1520) video footage at 70fps. To ensure that the footage is free of motion blur, we captured the scene at a high frame rate and additionally used tripods and a gimbal.

Motion capture

The cultivation of mastic was recorded in three days, from September 11th to 13th 2019, in Chios, Greece. Due to the nature of the cultivation process, each motion was split into different recordings. This resulted in separate motion files for each part of the process. In general, the cultivation of mastic was recorded realistically. However, in the actual cultivation and harvesting process, all tasks are days or weeks apart and usually take hours to be completed. As such, the expert had to perform a brief example of the gestures while still being realistic.

Post processing of Digital Assets

Ethnographic documentation

Video analysis and identification of key operational sequences that signify craft actions and processes. Definition of process flow diagrams for the weaving processes.

MoCap segmentation

After the correction and noise removal from the MoCap data, the data was segmented by gestures. The recordings initially were done by tasks, being one recording a whole task, these recordings were afterward segmented by gestures. One task could have only one gesture that is repeated several times or might have more gestures that are repeated through the task.

Examples of segmented MoCap data visualisation are presented in the following video:

Digitisation Outcomes

Villages

In the context of this research work, relevant context included the mastic villages formulated in a way to support the productions and economy of mastic cultivation and the industrial processes supporting the creation of mastic products. As such 3D context representations were acquired and implemented. For the reconstruction of the mastic villages of Chios, aerial images were acquired via a drone overlooking the Pyrgi, Mesta, Olympoi, and Elata villages on Chios, Greece. The subjects were large building complexes that pose several challenges due to the complex, medieval city planning, and fortification.

Mastic Museum

Mastic Cultivation Tools

The modeling of mastic cultivation tools was implemented from scratch in 3D using the Blender 3D

Mastic Factory Machines

For building the 3D models of the machines to inhabit the implemented virtual model, the actual machines exhibited at the Chios Mastic Museum were scanned using a handheld tri-optical scanner and photographic documentation was acquired. The scanned models were post-processed, using the Blender 3D software, to deal with known scanning issues relevant to metallic reflective materials, glass parts, etc.

Relevant Publications

Open Datasets

Applying the Mingei protocol in the Mastic pilot

STEP 2. Knowledge Elements

In this step, basic craft knowledge or otherwise knowledge elements are formed and digitally represented. In plain words, this means that the digitisations are entered in the Mingei repository, properly classified and linked to the other entities in the repository. Collectively, the classifications and links formed in this step form the semantic metadata that have been previously introduced.

Forming a knowledge element requires a comprehensive understanding of the digitally represented asset and is regarded as a digital curation process. Digital assets are represented in the system by knowledge elements associated with the result of a curation process that yields metadata, annotations, and descriptions. In this process, the original meta-data (if any) that accompany the digital asset are useful.

Representation of knowledge on craft processes includes curated material which is also digitally represented. The basic elements of such representations are the tools, machines, workshops, etc. At the same time of importance is the social and historical context.

Social and Historical Context

Persons

Human resources are invaluable in the description and explanation of craft practice and context. In Mingei the profession, communal activity, and traditions of a place are of close relevant to the contextualisation of a craft. Of relevance is also the participation of persons in communities, guilds, social or artistic movements. In our case , we record and represent basic information on the life and times of a person, such as occupation, places of visits, time and place of life and death, and key events relevant to the topic of study, such as education, professional activity, collaborations with other persons and so on.

Places

Places are represented with 2 coordinates, latitude and longitude. Regions are polygonal representations upon the surface of the Earth. We also use a data base of location names (GeoNames).

More refined representation of locations may require addresses or even arbitrary user-defined locations on Earth that are not included in the database. The coordinates of these locations are provided through the digital curation process and supported through a GIS system.

Events

We consider events as the changes of state in cultural, social or physical systems brought by phenomena or influenced by other events. Thus, the definition of an event requires at least two observations of this state in time.

Events are basic elements of stories in general and, in particular, to the type of “stories” we wish to capture in our craft representation. Knowledge elements of the event type are employed to represent:

  • Socio-historical context relevant to the expression of a craft instance.
  • Actions in craft processes

Craft Context

Tools

Equipment

Digital assets

The most relevant digital data types are:

  • Text
  • Photographs
  • Video
  • Audio
  • 3D reconstruction
  • Human motion digitisation

Additional digitisation types of humans, matter, and actions (e.g. thermal, multispectral, X-ray), provide measurements of more material properties that photography can capture. Such properties are surface reflection and transmission properties, chemical composition, its degree of viscosity at a certain temperature, and other.

We provide some examples of primary digital assets from the Mingei dataset

  • Digital text of curated text, literature, and testimonies.
  • Photographs: historic photographs, photographic documentation of artefacts, printed matter, records.
  • Audio: interviews of glassmaker and mastic producer.
  • Video of craft practice.
  • 3D reconstruction of a workshop from a laser scan.
  • 3D reconstruction of an artefact from a handheld scanner.
  • Motion capture of craft practice.

Photographs

3D Digitizations

Videos

Motion Vocabularies

3D Reconstructions

Motion Vocabulary Items

Relevant Publications

Open Datasets

Applying the Mingei protocol in the Mastic pilot

STEP 3. Craft Representation

In this step, the individual entities represented in the previous steps are linked to each other into an organic representation of the craft instance. This linkage implements the semantic representation of the craft.

The scope of this representation covers the following craft dimensions:

  • Tangible elements, such as materials, tools, and products
  • Craft actions and processes
  • Contextual knowledge that provides an understanding of
    • Artefact usage
    • The CH of a region and its people embedded in the artefact
    • The historic, geographical, economic, and social dimensions of the associated craft instance.

Social and Historic context

  • Representation of Fabulae

    Regards the structuring of ‘Events’ in MOP to ‘Fabulae’ presenting the sequence of ‘Events’ that are the backbone of a ‘Narrative’. Connection between ‘Events’ is established by two kinds of relations:

    1. Mereological relating events to other events that include them as parts, e.g., the invention of the flying shuttle is part of the life of the Industrial Revolution.
    2. Causal dependencies, relating ‘Events’ that in are predicated to have a cause-effect relationship, e.g., “the Industrial Revolution resulted in a reduction of the number of weavers”.

    Through MOP ‘Event’ entries are transformed into ‘Fabulae’.

Craft Representation

To encode craft understanding, activity diagrams are borrowed from Unified Modeling Language (UML) and used in the following sense. While UML represents computational actions that transform data are represented, in this work, physical actions that transform materials are. The transition types Transition, Fork, Merge, Join, and Branch are adopted and denoted as in UML.
Activity diagrams can be refined hierarchically, allowing the increase of representation detail at later stages. Moreover, their visual nature was found to support the collaboration with practitioners. Activity diagrams are essential in representing transitions between process steps, in which distinct tools are employed and actions take place. and, in particular, decision points and parallel tasks in the process.
The progression of sequential steps is modeled by a Transition link. Forks are used to represent the initiation of two parallel tasks. In Merge transition, two or more control paths unite and Join connects steps that should be completed before the transitions to the next step. Merge and Join transitions are structurally similar, but a join is a synchronization across a set of parallel flows, while in a merge only a single flow is active. Finally, Branch transitions connect a step with a decision step that accepts tokens on one incoming edge and selects one outgoing alternative. Branch nodes control the flow of a process by selecting one of several alternatives, based on the outcome of a condition evaluation.

Process schemas

Activity diagram are transcribed into transition graphs. The MOP UI facilitates the structured instantiation of process schemas and their steps. Data fields are used to enter appellations, informal descriptions, and step order. Transitions are instantiated via dynamic UI components that adapt to transition type.

Processes

Processes representations account for the events that took place during the execution of a process schema. Intuitively, a process is an individual flow of events, out of those possible in the activity diagram. The MOP UI enables the instantiation of processes representation, via the entry and chronological ordering of events, accompanied by the recordings that document them.

Relevant Publications

  • Digitisation of traditional craft processes, (in press, JOCCH-21-0060), X. Zabulis, C. Meghini, A. Dubois, P. Doulgeraki, N. Partarakis, I. Adami, E. Karuzaki, A. Carre, N. Patsiouras, D. Kaplanidi, D. Metilli, V. Bartalesi, C. Ringas, E. Tasiopoulou, Z. Stefanidi, ACM Journal on Computing and Cultural Heritage, DOI:10.1145/3494675.

Open Datasets

Applying the Mingei protocol in the Mastic pilot

STEP 4. Narratives

A digital narrative or, simply, a narrative is an abstraction that represents a set of facts that have happened in the world. These facts are connected to each other in a way that makes them a story.

This abstraction includes a narration, which is the way that a certain author has told the story. Individual narratives may be focused at a subject and include a pertinent subset of the events that are represented in the fabula.

The narration tells the story, using a certain medium. In Mingei multiple media are employed, starting from verbal and visual and reaching up to immersive and interactive narrations. A medium, and correspondingly, a media object may be multidimensional; for example the audio and visual components of a mobile phone “video” recording, or the combination of video and 3D MoCap recordings.

A narrator is a person or a program that presents a media object, i.e., verbally, by visual rendering, sonification, VR, etc. Besides multimedia and immersive presentations (discussed in the next step), conventional, verbal and visual, media objects and channels are of fundamental importance. A textual media object can be exported as a narrative for a person or an avatar to read, and a formatted document with illustrations, or an “infographic”, can output channels. In this context, Mingei aspires the export of more imaginative and interactive uses of conventional verbal and visual outputs, including scripts for theatrical re-enactment of stories and role-play on contextual dimensions of craft instance, such as skill, trade, and labour. The choice of narration medium is relevant to the use of the content, the intended audiences, and the experiences to be rendered these are discussed in the next step. In this step, we focus on the narrative representation.

A narrative can include additional information besides the connection of the narration to the facts that are narrated. In particular, a digital narrative consists of three main elements:

  1. A representation of the fabula made of representations of the events that the narration is telling.
  2. One or more media objects.
  3. A digital representation of the reference function that connects the (representations of the) events of a story with a fragment of a media object that describes them and allows to derive the plot.

Each narration of a fabula employs the use of one or more media objects, which have been authored by one or more narrator(s). Through the Mingei Online Platform, a tool for authoring and visualising narratives is provided.

Relevant Publications

Applying the Mingei protocol in the Mastic pilot

Step 5. Information tools

Historical narratives

Virtual Humans

For the needs of the demonstrations 3D virtual humans, with their corresponding garments and accessories, have been carried out based on cultural and historical information and sources. VHs were created using references from the ethnographic research. For the definition of the 3D meshes and the design of the skin surfaces a customized toolchain was employed that combines automatic generation methods, with manual editing and refining. The approach utilizes VH creation software and 3D modeling software for clothes and accessories. The final model inherits and encapsulates the required structural components, such as bone and skin attachment data. This allows the VH to be dynamically animated and to exhibit real-time animation capabilities.

Another important aspect for attaining a high level of realism is the motion, behavior, and natural interaction of the VH with users, to avoid the “uncanny valley” effect. The VH must have the ability for verbal as well as nonverbal communication skills, be intelligent, have natural communication with the users, perceive information from the user, and physically react with them. The deformation of the skinned characters during their movements must be realistic, smooth, and not contain any discontinuities.

Garment design was also considered during the implementation of avatars. To ensure appropriate rigging, since the generated models are automatically rigged an additional checking was performed to ensure that the rig is applied correctly, and the bones are well adjusted to the 3D body.

An example of a Virtual Human for the mastic pilot can be seen in the following video:

Hello! My name is Anna. Welcome to the chewing gum factory of the Chios Gum Mastic Growers Association.

I am coming from the village of Tholopotami, just a 25 minutes ride by car from the factory. I grew up and still live in the village. Both my parents come from the village and they are cultivators of mastic, olive, and almond trees. We always had an immediate income from the almond production because it does not need any processing, in contrast to mastic that needs year-long processing before we deliver it to the Association. I also have two siblings and when we were all children, we used to accompany our parents in the fields. Nevertheless, I only finished primary school because I was mostly cultivating with my parents.

Video documentation of the craft

Motion Re-targeting

The processing of the motion data has been done on Autodesk MotionBuilder software which is dedicated to animation and the direct integration of motion capture technologies. The process requires different steps: (a) creation of an “actor” in MotionBuilder with skeleton definition corresponding to the BVH hierarchy, (b) mapping of the received animations on the actor, and (c) synchronization of the VH with the actor by adjusting the 2 models so that the measurements match and the animations are correctly reproduced. Examples of the outcomes are presented below:

Access the full playlist from the this link.

Applying the Mingei protocol in the Mastic pilot

Step 6. Experiences

Narrations in the Mastic Factory

The installation at the mastic factory exhibition room is comprised of four tablet devices mounted on floor mounter bases that are located in four main spots of the museum. From each tablet, a specific area of the museum is covered and augmented through the camera of the tablet with hot spots. In each hot spot, one or more stories to be told exist. By selecting the hotspot, a VH appears that is the visual twin of a persona used to work in the factory and operating one of the machines in visual approximation to its location. When the hot spot is selected the VH appears in the factory through the camera of the table to narrate his life story and his daily life and work at the factory.

Craft presentations in the mastic field

One of the requirements of presenting the craft was to display its seasonality. Very often visitors do not understand the complete process of mastic cultivation. Another necessity for the museum was the exploitation of the external spaces and the beautiful mastic tree field which is often overlooked by the visitors. It became also evident that this “guided” tour (it is a simple and straight path through the phases of mastic cultivation, harvesting, and cleaning) is a perfect example of the yearlong process. Visitors of the rural space outside the museum can experience mastic cultivation in the field through their mobile devices. The application facilitates an AR-capable device to recognize metallic sculptures that exist in the rural space of the museum. Through the camera, these sculptures become alive to present typical cultivation activities. The following figure presents an example of app screens with an emphasis in the middle on the AR augmentation of the sculpts that are part of the mastic field with animations of the cultivation process.

Craft Training

Craft training is intended for demonstrating the cultivation activities to the visitors of the museum thus providing a more immersive experience. This application was installed on the ground floor of the museum in the multimedia space. The installation is comprised of a personal computer and a monitor together with a depth sensor for tracking the user’s actions. The user stands in front of the installation and follows the instructions provided on the screen to mimic craft actions. An example of this process is presented in the following figure.

Presentation of geographic context

Information on geographical location and context shows environmental aspects affecting craft practice and development. We developed Airborne an immersive flight simulator allowing users to fly over various mastic villages of Chios. During the flyover, users can stop at each village and retrieve multimedia and text information related to those villages. Airborne is installed in the multimedia room of the museum. The setup was very simple and straightforward as it involved a desktop computer set. There are two options available (a) automated tour and (b) flight simulator. The automated tour targets users that wish to explore the mastic villages in a movie-like way while the flight simulator is more gameplay-oriented since users have control of the virtual drone flying on Chios sky and are free to explore information in any way they like. In the future, the setup will be updated with a large touch-enabled screen to enhance the gameplay of the installation.