Conceptualization and semantic description in digitized tangible/intangible content

Abstract

The dissertation investigates the visual interpretation of heritage objects and practices providing both a theoretical framework for understanding the process of meaning-assignment and a formal ontology for grounding propositions about visual objects into a constituent-based framework. The thesis starts its investigation from the theory of the perception of heritage, taking a semiotic standpoint and proposing a new framework of understanding Jakobson’s Cultural Model of Interpretation (JCMI). The model, based on Jakobson’s model of communication, illustrate the nuclear components used in the process of interpretation of our heritage. JCMI focuses particularly on the reconstruction of the original meaning, providing to the reader a theory of interpretation as a way to understand how the meaning of the object is constructed. Building upon the work of Eco, the dissertation advances a functional theory for the classification of the perceptual experience, correlating perceived signals to situations and physical things. The theoretical work analyses the use of the correlation in three diverse scenarios: recognition of the known, enhancement of the known and identification of the unknown, outlining how we use diverse similarity-based recognitions for defining the constituents of a perceptual manifestation. The resulting structure helps to examine the meaning assignment of cultural content and to study the framing of both physical objects and performative actions in our everyday visual experience. The broad relevance of the framework in the context of the heritage practice is made explicit through the analysis of the problem of instance vs type, which is illustrated using pictorial examples of a famous subject of Eastern and Western art: Saint George. Instances of the visual representations of the saint are used to demonstrate both the function of the theoretical framework as well as its alignment with some iconographical representation theories in Art History, specifically Panofsky’s and Van Staten’s image description frameworks. These two theories are integrated and grounded within Jakobson’s Cultural Model of Interpretation, highlighting the necessary correlations and critiques. The framework developed is then used to guide the formalisation of an ontology which is constructed as an extension of CIDOC-CRM and sustain the recording of statements about the diverse elements present in a visual representation on the base of their interpretation. The result, tested with artworks coming from the church of Asinou, a small Byzantine foundation in Cyprus, demonstrates the capacity of the formal ontology to sustain the recording of statements about the denotative and the connotative dimension of the church’s iconographical objects, helping unveil their contextual meaning through the clarification of symbols and constituents, liturgical performances and historical influences. The ontology is further linked and harmonised with 3D data annotations structures, providing a framework of operation for using the ontology to classify visual annotations within a three- dimensional environment. The core components of the standard annotation model, the W3C Web Annotation ontology, are presented and analysed with respect to their usability for 3D annotations, specifically examining the connection between object fragment and annotation. The fragment-object relation is scrutinised using examples of solutions used by 3D web annotation projects. Moreover, a proposal for the creation of a shared URI structure able to characterise single areas within a scene is outlined. The Web Annotation model is further tested against 3D data and used to provide possible information pipelines for the semantic annotation and interconnection of 2D-3D areas within the Aioli project. Additionally, supplementary uses of the ontology (e.g. data harmonisation and enhancement, semantic reasoning) are examined and presented. After having provided and tested the feasibility of the ontology to document the meaning- assignment in visual work, the dissertation further elaborates on the validity of the recorded information, specifically the one grounded on digital mediations such as photographs or three- dimensional hyper-realistic reconstructions. In order to tackle the problem, a provenance-based framework for the documentation of reality-based recording is presented. The framework provides a formalisation for recording the different steps of a digitisation workflow in order to document the accuracy and reliability of the digitisation as well as to frame the developed digital objects as scholarly product and not mere visual artefact. The framework analyses both the essential processes to be carried out and the metadata to be gathered in digital photogrammetry survey projects, breaking up a typical digitisation workflow into seven iterative and repeatable steps. The documentation of each step is achieved using key metadata registers that capture the main framework of data interactions, starting from the functional requirements until the scientific analysis of a digital documentation project. The provenance framework is tested using the digitisation of a part of the church of Asinou as key study, and the output produced is modelled using CIDOC-CRM and extensions. The results are linked with the annotation model and the ontology to provide an example of a full semantic documentation of a digitised heritage object which includes and records its digitisation, data processing, three-dimensional annotation and cultural interpretation of that object. The result is a single graph of object, practice and meanings linked together.