Introduction

There is mounting evidence1,2,3,4,5,6 that emotions are connected to climate change in many ways and shape humans’ multifaceted responses to the climate crisis. Analogously, emotions play a significant role in evaluating the built environment and calculating building sustainability rankings. A study performed by Curtis et al.7 confirmed the direct and positive relation between the positive emotions of consumers and their purchase intent. The United Nations8 has proposed paying more attention to emotions when dealing with the 17 Sustainable Development Goals of the 2030 Agenda.

A comfortable home has so far mostly been assessed in terms of physical, for the most part thermal, comfort and studied through the lens of natural and building sciences with a focus on technical approaches and laboratory settings. Yet physiological comfort, such as the right temperature, is not the only aspect people need and expect from their homes, as confirmed by many studies by philosophers, geographers, sociologists, historians, and anthropologists trying to determine the meaning of a home and what makes a certain place a home. Social, emotional, and cultural meaning represents another layer of a home, which is perceived by its inhabitants as a place for family, involving control, rest, and security, without disruptions, and thus a significant part of their well-being9.

Looking at comfort from the perspective of different disciplines, an overview by Ortiz et al.10 presents it as a subjective and multidimensional phenomenon with a biological component, reflected as a physiological, emotional, and behavioral response to environmental stimuli. Comfort, therefore, has links to health and well-being10. Besides physiological reactions, subjective reactions to stimuli present in the environment are also part of the sense of comfort that can indicate harmony and neutrality with the stimuli. These reactions can be psychological, physical, social, behavioral, and physiological. As humans react to stimuli in their environment, their emotions guide their behavior, meaning that the emotional reaction comes before the behavioral one. When their emotions are negative (displeasure, discomfort, or stress), people behave in ways to reduce or eliminate any negative stimulus present in the environment10.

Many studies theoretically and practically analyze how different features of a property and its context contribute to a human buyer’s overall emotional response and how that translates into hedonic and perceived values (Fig. 1).

Fig. 1
Fig. 1Fig. 1Fig. 1
Full size image

Behavioral finance and emotional psychology theories, the impact of emotions on investment decision-making, emotion-aware design and cross-cultural analysis.

Figure 1 suggests that the connection between emotions and real estate value is intricate, as emotions affect how individuals view and engage with environments, ultimately shaping their readiness to invest in them. The studies described in Fig. 1 indicate that positive feelings can increase real estate values, whereas negative emotions may discourage potential buyers. Additionally, the emotions of prospective buyers can affect how they view real estate features and their general assessment of specific real estate. Although logical aspects such as location and real estate features are significant, emotions frequently have a considerable influence on property investment options, occasionally surpassing logical factors. Grasping the emotional factors that influence investors and buyers can enable property experts to determine real estate value and ensure successful deals more accurately (Fig. 1).

De Looze et al.83 also looked at the way comfort is defined across disciplines and determined that the definitions share three elements: that various factors affect comfort (they can be psychological, physical, or physiological), as well as that comfort is a subjective and personal experience and is a reaction to the environment.

Studies have shown that both health and longevity can benefit from positive emotions84. Evidence suggests that positive emotions may contribute to lower blood pressure, stronger immunity, healthier cardiovascular and neuroendocrine systems, and reduced inflammatory processes, whereas negative emotions can cause depression, anxiety, stress, and lead to damage in the cardiovascular system85,86,87,88.

Occupants often want healthy and comfortable working spaces that are conducive to refreshment and relaxation, thus well-being in office buildings is a particularly important concept. Storey and Pedersen89 noted that the indoor environment encompasses the full range of aspects that contribute to well-being, including physical ones such as appropriate lighting, indoor air quality, and thermal and acoustic comfort, as well as intellectual and emotional ones such as opportunities for personal control, interior design, and engagement with nature90. Companies install relaxation areas or gaming spaces in an attempt to make the workplace an enjoyable space91 and their goal is not limited to providing employees with a space to enjoy during the working day; they also want to persuade employees to see the office as a general location for relaxation and pleasure and stay there instead of going elsewhere to relax. Even when work consists of mostly routine tasks, a workplace environment that is fun and attractive can encourage emotional attitudes that can contribute to maximum employee commitment and help control employee behavior92. Supplementary Materials 1 describes green rating systems and real estate valuation, emphasizing emotions and sentiment analysis.

Existing literature (Fig. 1) suggests that human emotions predict investment value; however, there is limited understanding of the specific elements of human MAPS, context, and perceived rental value that influence this relationship. The purpose of this research was to examine this issue, develop the MOVE and MAPS models using 879 million data points on the emotional and affective states of depersonalized passers-by collected from November 2017 to January 1, 2023.

Results

The creation of the Multimodal Property Video Neuroanalytics was a two-stage process spread over two projects. The first stage was carried out during the European Union H2020 ROCK project to build the Video Neuroanalytics and its related smart infrastructure. The project involved tracking people at 10 places through Vilnius and produced more than 879 million units of depersonalized passersby’ emotional, affective, physiological, and performance and city environment etc. Building on the outputs produced by the ROCK project, the second stage involved the creation of the Multimodal Property Video Neuroanalytics method and system during the AFFECTS and META-MUSEUM projects.

Case study 1: Human diurnal rhythms, correlations and models in vilnius city

The field research phase of the ROCK project, a joint effort by Vilnius Gediminas Technical University (VGTU) and Vilnius Municipality, involved mounting the sets of our equipment subsystem at six intersections in Vilnius city and on two buildings. Figure 2 shows an image of the MOVE equipment fragment in Gediminas Avenue.

Fig. 2
Fig. 2
Full size image

A fragment of the MOVE equipment in Gediminas Avenue, Vilnius, Lithuania.

MAPS data are collected in Vilnius City Municipality building, the Business Center at Narbuto Street 5 and at six intersections in Vilnius: (1) the intersection of Pamėnkalnio Street, Jogailos Street, Islandijos Street and Pylimo Street; (2) the intersection of Santariškių Street and Baublio Street; (3) the intersection of Žygimantų Street and T. Vrublevskio Street; (4) the intersection of Šventaragio Street and Gedimino Avenue; (5) the intersection of Šventaragio Street and Pilies Street, and (6) the intersection of Kareivių Street, Kalvarijų Street and Ozo Street. During the summer, MAPS data were also collected at three beaches in Vilnius.

The equipment subsystem sets were used to collect physiological and biometric data and comprise devices that can track temperature (infrared camera FLIR A35SC), emotions shown by facial expressions (FaceReader 7–9), respiration rate (sensor X4M200), people flow count (H.264 Indoor Mini Dome IP Camera), eye pupils (Mirametrix S2 Eye-Tracker), voice emotions (QA5 SDK), heart rate (iHealth Wireless Blood Pressure Monitor) and brain signals (Enobio Helmet). The MOVE can be managed using a user interface with user-friendly controls. Between November 6, 2017 and January 1, 2023 anonymous passersby were observed tracking their affective state, emotional state, physiological state, arousal and valence (MAPS) at eight locations across Vilnius. These remote data were collected in real time gathering them in different formats and in three layers and then processed, integrated and analyzed. The Multimodal Property Video Neuroanalytics (MOVE) was used for the gathering and more than 879 million of such MAPS data were collected over the period of tracking. The data offer a fairly accurate picture of how people experience a city and how they feel there. All data were gathered using video streams, except for the respiratory rates of passersby. Respiratory rates were tracked using X4M200, a non-contact respiration sensor with the technology of electromagnetic signal reflection to gather information about a target.

This research was undertaken in a real-life setting and led to results that match the natural patterns characteristic to human diurnal rhythms. The results of this study, including approximately 879 million MAPS data points, correlate with the Twitter-based worldwide data covering Canada, Australia, the UK, the USA, Asia, Europe, Latin America, North America, and Oceania93. Numerous scholars have noticed that a necessary intensity of average wind speed, O3, weather temperature, happiness, relative humidity, etc. also depend on the hour of the day. SPSS, a statistical software suite, was used to analyze the data points, which totaled more than 60 million in all. Figure 3 describes the hourly average wind speed, O3, weather temperature, happiness, and relative humidity average data recorded on weekdays in Vilnius city.

Fig. 3
Fig. 3Fig. 3
Full size image

Average wind speed (a), O3 (b), weather temperature (c), happiness (d), relative humidity’s (e) hourly average values on weekdays. Values tracking changes seen in people passing by at 10 locations across Vilnius were gathered every second to calculate hourly average valence. On each weekday, these values were accumulated at 95% confidence intervals. Each day of the week is highlighted in a specific color. The x axis shows the time start at midnight and the y axis displays the medium valence for each day of a seven-day period. The valence scores vary between − 1 and 1.

The research findings Kaklauskas et al.94 determined on the basis of 208 million data points on diurnal emotions, facial temperature, and valence, for instance, are similar to the findings presented in a research of “X” (former Twitter) content related to this area and spanning many regions and countries.

The study found numerous significant correlations between the factors analyzed. Table 1 presents some of the many correlations obtained.

Table 1 Table presents several of the many significant correlations between the analyzed MAPS, pollution, and meteorological (MAPME) factors identified during the study.
Full size table

Our experiments aimed to gather data for MAPS state models (Table S1). Separate MAPS models are created for employees under analysis. The results of the correlation analysis between pollution (SO2, PM2.5, PM10, NO2, CO, O3) and magnetic storm on the one hand and MAPS states on the other are presented in Table S2.

We applied the data calibration of collected data to align with known standards to ensure accuracy, consistency, and comparability across different sensors or over time. It involves fine-tuning data from a specific sensor and survey to match another, correcting systematic errors, and validating MAPS models outputs against real-world data to make them reliable and trustworthy.

The research aim was to determine the relationship between the MAPS, pollution, and meteorological (MAPME) parameters. Dependent variables were found to be the MAPS, and independent variables – various pollution, meteorological, and emotional parameters. The process of data collection and processing was carried out as follows:

  1. 1.

    Data sources. By applying the reliable, scientifically based equipment subsystem, we collected 879 million anonymized data on emotional (disgusted, sad, happy, scared, surprised, neutral, angry) states, valence, arousal, affective parameters (interest, confusion, boredom), and physiological states (breathing rate and heart rate) (MAPS) from passers-by and employees. The research was performed from 2017 at eleven places in Vilnius city. Data on pollution were received hourly from the Lithuanian Environmental Protection Agency, while the data on temperatures were provided every hour by the Lithuanian Hydrometeorological Service. These parameters were collected through proven and reliable sensors, ensuring high data quality.

  2. 2.

    Relationships between variables. After conducting a correlation analysis, it was found that the MAPME parameters of emotions are interrelated. Table 1 presents several of the many significant correlations between the analyzed factors identified during the study. Therefore, when performing linear regression analysis for the arousal, valence, emotional (anger, disgust, happiness, neutrality, sadness, fear, surprise) and affective (confusion, boredom) states, the other MAPS, in addition to the pollution and meteorological parameters, were used as independent variables.

  3. 3.

    Data imputation. To ensure the completeness and quality of the data set, missing data were imputed using the predictive mean matching method.

  4. 4.

    Models development. The collected data were divided into two parts – training (80%) and testing (20%) sets – to ensure the reliability of the model and the ability to generalise the results.

  5. 5.

    Regression analysis. Linear regression analysis was performed for the MAPS under analysis. The regression models for the diurnal valence, arousal, emotional (anger, disgust, happiness, neutrality, sadness, fear, surprise) and affective (confusion, boredom) states were developed and assessed using R2, the standardized coefficients β, the p-value, and the elasticity coefficient. The justification for weight assignments in the evaluation system (MAPS regression models) was performed by applying beta (β) weights (standardised regression coefficients), which show the size and direction of each predictor variable’s effect on the MAPS states.

These detailed steps ensure that the research results are obtained from data of the appropriate format and quality, which allows for more accurate predictions from the selected regression models.

In order to prevent observer bias, we triangulated our data with different data collection methods and sources, ensured inter-rater reliability, and recorded the data consistently. With triangulation, we used multiple data collection methods to study the same phenomenon. We calibrated our methods and data collection sensors regularly to keep inter-rater reliability high. For example, in the study conducted, the FaceReader research software solution (Versions 7–9, Noldus Information Technology95) was used to collect and analyse MAPS data. This technology is widely recognised in the scientific community – it is reliable, and is used in thousands of studies worldwide. FaceReader is based on rigorous methodology and validated by scientific data, so it contributes to reducing the risk of algorithmic bias.

In order to prevent observer bias, we triangulated our data with different data collection methods and sources, ensured inter-rater reliability, and recorded our data consistently. Sampling bias, the Hawthorne effect (observer bias), ascertainment bias, undercoverage bias, and other96 biases did not occur because we analyzed 879 million depersonalized passer-by data points. Consequently, the sample reflects the entire population accurately. Omitted variable bias occurs in our regression models because they do not include perceived rental value variables between November 2017 and January 1, 2023. Figure 1 analyses how the property and its context impact emotions, the impact of these emotions on investment decision-making and related cross-cultural analysis. For example, the same sun shines on all investment alternatives analyzed and similarly influences all people. Therefore, a tourist’s happiness on a sunny day equally contributes to the data of the alternatives under analysis, but the person’s emotional state has no direct bearing on the rental value of an adjacent office building for a long-term corporate tenant.

We used data calibration, which is the process of adjusting raw or collected data to align it with a known standard or desired model to ensure accuracy, consistency, and comparability across sources or over time. It involves fine-tuning data from a specific sensor or survey to match another, correcting systematic errors, and validating model outputs against real-world data to make them reliable and trustworthy. During the study, calibration of the equipment subsystem measuring instruments was performed. As part of the procedure, the readings of the measuring device were compared with a standard – a measuring instrument with a known and confirmed accuracy. This ensured the accuracy and reliability of the measuring instruments.

An additional sensitivity analysis was performed to confirm the overall reliability of the model. It was found, for example, that with an increase in the value of the magnetic storm variable by 1 per cent, the value of the angry variable increases by 0.2 per cent. Meanwhile, when the average wind speed variable decreases by 1 per cent, the values of the happy variable increase by 0.09 per cent, which confirms the sensitivity of the valence variable to changes in the happy variable.

Global research shows that emotions are an important factor in assessing the environment and buildings. To address the 17 Sustainable Development Goals, the UN member countries are proposing a greater focus on emotions.

Global research as well as our own shows that the emotions of passers-by reflect the general mood of the location being analysed and can be an indicator of the general mood:

  • The study reveals that emotions depend on the context that describes what is happening around a person. This means that context factors directly influence the emotions of passers-by.

  • In the study, passers-by are treated as a sample of potential consumers and investors, and their emotions are recorded in specific locations (near the subjects being evaluated). This makes it possible to evaluate the overall impact of the location on the emotional state that influences decisions.

  • The data required for the study, which were collected from a large number (over 879 million) of anonymous data sources, make it possible to determine general moods and trends that reflect not the mood of a single individual, but the overall impact of the environment on individuals who are near the subjects being evaluated, which means that they directly interact with the subjects. This is not an evaluation of the mood of one person (tourist), but a statistically significant sample of all people passing by.

The MOVE system combines the MAPS data of passers-by (emotional, affective, and physiological states) with the details of the building and its context, creating a holistic map of investment appraisal. The developed methodology makes it possible to calculate the perceived value of investments, taking emotions into account. For example, it was calculated that for a gym to become competitive, its perceived rental price would need to increase from EUR 84/m2 to EUR 139.43/m2 in order to obtain the required valence value. This example confirms that differences in real estate values can be explained by the impact of the emotional and affective states of passers-by, which indirectly increases the sales value. Emotions are therefore an important indicator in determining the true value of a project.

We employed a stepwise regression approach for the incremental iterative development of MAPS states regression models, which includes the selection of independent variables for inclusion in a final model. It entails consecutively eliminating possible explanatory variables and checking for statistical significance following each step. With the help of the regression subsystem, MAPS states models were created using stepwise regression. Thirty MAPS, pollution, and meteorological (MAPME) variables were analyzed to create valence, arousal, emotional, and affective states (MAPS) models (Tables S1). The MAPS data was divided 80/20, allocating 80% for training and the other 20% for testing. The elasticity coefficient, the standardized beta coefficients β, the p-value, and R2 were the model metrics used for our systematic assessment of how well the valence, arousal, emotional (angry, disgusted, happy, neutral, sad, scared, surprised) and affective (confusion, boredom) states regression models perform.

Table S1a shows the original 11 MAPS models developed during our research. In the table, all statistically significant correlations (p < 0.05) are highlighted in italic. The table also includes the standardized β coefficients that measure the impact of a specific independent MAPME variable on the dependent variables of valence, arousal, emotional and affective states (higher absolute β values indicate bigger impact). The original 11 MAPS models, on average, explain 86.9% of the variance seen in the models of anger (R2Angr = 0.99), valence (R2Val = 0.997), arousal (R2Arous = 0.886), and happiness (R2Happ = 0.994) caused by changes in the independent MAPME variables. This rate means they are suitable for analysis (p < 0.05, Table S1a).

The independent MAPME variables with no statistically significant effect on the dependent variable were all removed from the 11 MAPS models thus creating adjusted versions of the models. These adjusted versions are suitable for analysis (p < 0.05). The validation of the adjusted anger (R2Angr = 0.9879), valence (R2Val = 0.9955), arousal (R2Arous = 0.845), and happiness (R2Happ = 0.9925) models determined all of them to be suitable for analysis and on average 81.3% cases of the distribution seen among the values of the dependent variables were explained (Tables S1b). The studies presented in Fig. 1, Table 1, and 11 MAPS models (Table S1), which were developed from 879 million data points, show that the emotions of random passersby are valid and can be a direct proxy for the valuation decisions of property investors and tenants. The study analyses investment alternatives (a shopping center and a gym) where these passersby may frequently visit or even live (an apartment complex). The population of passersby will not necessarily be demographically and emotionally distinct from the population of potential buyers/tenants.

We used standardized β coefficients to measure the impact each individual MAPME variable makes on a specific person’s 11 MAPS states and determined that multiple MAPME variables make a noticeable impact and thus changes in them can improve a person’s MAPS states (Tables S1b). In the happiness model, valence (β = 1.246), angry (β = 0.465), sad (β = 0.21) and neutral (β = −0.179) were the MAPME variables with the biggest impact on happiness and changes in their values best explained the variance of happiness.

The elasticity coefficient was calculated for each independent MAPME variable included in the adjusted MAPS models to measure the effect that changes in these independent variables make on the dependent variables of MAPS states. Table S1b shows the results of these calculations. Furthermore, Table S1b shows that an average improvement of 1% in the MAPS, pollution, and meteorological (MAPME) variables results in an improvement of 0.282% in the person’s happiness, of 0.473% in the person’s valence, and of 0.167% in the person’s arousal (Tables S1b).

Regression analysis was performed and its results show that the models of the dependence between the dependent MAPS variables and the independent variables is satisfactory and fit for analysis (p < 0.05) (Supplementary materials 2). Fig. S1 shows the dependencies between valence and pollution ((a) PM2.5, (b) NO2, (c) PM10), and the (d) apparent temperature calculated using Eq. 1, (e) a magnetic storm, and (f) atmospheric pressure. Fig. S2 shows the dependencies between arousal and pollution ((a) SO2, (b) PM2.5, (c) PM10), and (d) atmospheric pressure at the station level calculated using Eq. 2, (e) a magnetic storm, and (f) atmospheric pressure.

Our research produced more precise mechanistic explanations linking emotional states to valuation outcomes (hedonic and perceived values) as shown in Fig. 1. Figure 1 also presents various research in the field of emotions within urban and property contexts to show broader geographical/cultural representations and consider market dynamics. This broader picture and our research help us understand how to conduct this research in other countries, considering cross-cultural applicability beyond the context of Lithuania, and provide more actionable implementation strategies for emotion-aware design. Emotion-aware design encompasses the space’s shape (aesthetics, functionality), interior, cultural context, and other design elements, potentially enhancing the real estate’s attractiveness and value. Attractive interior design features can evoke specific emotions and create beautiful environment and an enhanced sense of well-being, increasing its hedonic and perceived values (Fig. 1).

Case study 2: Multicriteria and sentiment analysis of investment projects valued using the INVAR method

Calculating the utility degree and perceived rental value of the investment projects

A pleasant, aesthetic, and comfortable environment not only evokes positive emotional states during the life cycle of a building but can also influence the choice of real estate investment. An attractively built environment creates a sense of wellbeing and security for potential buyers, who are inclined to spend more of their time in an environment that meets their needs. The aim of our case studies was to assess the influence of system of building, its context and MAPS criteria and its weights on the outcome of the analysis (priority, utility degree, values, and evidence-based digital tips) of three specific investment projects.

For the case study, we installed the Video Neuroanalytics at various intersections in the city of Vilnius. The emotional, affective, and physiological states (MAPS) data scanning, collection, and transmission points were chosen based on close proximity to the investment projects. The Video Neuroanalytics data for the creation of the emotional and affective states of depersonalized passersby were collected between November 2017 and January 1, 2023. Around 879 million MAPS data points have been collected.

In order to select the best and most profitable project in terms of investment, three sustainable investment projects (a gym (Ia), shopping center (Ib), and apartment complex (Ic)) were chosen for the analysis (Fig. 4). Using information about the projects publicly available on the Internet, baseline data were collected for the analysis, and emotional (happy, sad, angry, and valence) and affective (surprised, scared, disgusted, and excited) MAPS data were collected with the Video Neuroanalytics. The collected data are presented in Table 2, which consists of the multidimensional building, its context and MAPS criteria, as well as their values and weights.

Fig. 4
Fig. 4
Full size image

Investment projects selected for this study: Ia – gym; Ib – shopping center; Ic – apartment complex.

Table 2 Decision matrix for the evaluation of the investment projects and the results of the multi-criteria evaluation.
Full size table

The first investment project under analysis, a gym (Fig. 4, Ia), is located close to the central part of Vilnius.

The second investment project, a shopping center (Fig. 4, Ib), is located in a building close to the central part of Vilnius, in the Naujamiestis district. The third investment project selected for the analysis, the apartment complex (Fig. 4, Ic), is located in the part of Vilnius east of the Old Town, near the Vilnia River, in the Old Town Eldership. Each potential project is described in more detail in Supplementary Materials 3.

The baseline data for the INVAR (degree of project utility and investment value assessments) method64 calculations for the selected investment projects including the gym (Ia), located at J. Jasinskio Str. 16, shopping center (Ib), located at Gediminas Ave. 28, and apartment complex (Ic), located at Aukštaičių Str. 18, are presented in Table 2.

The INVAR method was utilized with the integration of the building, its context and MAPS indicators. The analysis covered three different projects: a gym (Ia), located at Jasinskio Str. 16; a shopping center (Ib), located at Gediminas Ave. 28; and an apartment complex (Ic), located at Aukštaičių Str. 19. The decision matrix (Table 2) is divided into two parts and into four groups of criteria. On the right-hand side of the first part of the table the data include building and its context criteria. On the left-hand side of the second part of the table analogous criteria are presented with the same values and a fifth MAPS criteria has been added. The results of the analysis of the options are presented at the end of Table 2. Each criterion has been weighted according to investor requirements. The sum of the weights for all the criteria is set equal to one.

The multi-criteria analysis of the investment projects was carried out using publicly available databases, results from a survey carried out in the real estate sector, and the experts’ long personal experience (Table 2). In terms of the totality of the characteristics of the five groups of indicators on the left-hand side of the Table 2, the apartment complex (Ic) ranked first in terms of its utility degree (N3 = 100%), compared to the other two investment projects, of which the shopping center (Ib) was ranked second (N2 = 98.05%) and the gym (Ia) was ranked third (N1 = 84.05%), 15.95% lower than the apartment complex (Ic).

An identical matrix was constructed from the data on the right-hand side of Table 2 to demonstrate the influence of emotions on the results, so the emotion criterion groups have not been included here. In the calculation of steps 1–5 of the INVAR method, after taking into account the totality of the characteristics of the four building and its context groups of criteria on the right-hand side of the table, the priority rankings of the projects in terms of utility degree were analogous, i.e., the first place was taken by the apartment complex (Ic) (N3 = 100%), the shopping center (Ib) came second (N2 = 85.53%), and the gym (Ia) was third (N1 = 71.32%, 28.68% less than the apartment complex (Ic)).

These results suggest that, for appropriate assessment of the utility degree of the investments, the assessment of emotions plays an important role. If emotional parameters are not assessed, the value of the Sports club and Shopping center drops by 12,73% and 12,52% accordingly. The differences seen between the results of utility degree value analysis could be explained by the impact of the emotional and affective states of passers-by as they indirectly push the value of that sales up (Table 2).

The INVAR method is applied to the analysis of the perceived average annual sq. m. rental value of selected investment projects, and the results are presented below. We obtain the valence value by subtracting the highest negative emotion value from the happiness value. We made an extensive use of the FaceReader program for facial analysis in our research. This program can recognize valence, arousal and emotions (happy, angry, scared, surprised, disgusted, sad, and a neutral state). According to Noldus95(page 80), “the valence is calculated as the intensity of ‘Happy’ minus the intensity of the negative emotion with the highest intensity. For instance, if the intensity of ‘Happy’ is 0.8 and the intensities of ‘Sad’, ‘Angry’, ‘Scared’ and ‘Disgusted’ are 0.1, 0.0, 0.05 and 0.05, respectively, then the valence is 0.7.”

A potential investor participated in the study, whose valence data were collected during a view and discussion of the alternatives under consideration. The study used all the building and surrounding environment data (Table 2) and the valence data of the potential investor who participated (Table 3).

Table 3 Initial data for calculating the perceived average annual sq. m. rental value of the Sports club.
Full size table

In this stage, the aim is to calculate the perceived average annual sq.m. rental value of the Sports club to make the investment the average equal option of all investment options being considered. For example, the INVAR method was used to optimize the Sports club’s valence metric. The initial perceived average annual sq. m. rental value of the Sports club is equal to the average yearly sq. m. rental price (84 euro/m2). The task can be identified in this way: What is the perceived average annual sq. m. rental value of the Sports club that will make it similarly competitive in the real estate market, compared to the Shopping centre and Apartment complex under analysis? This can be calculated if a holistic evaluation of the alternatives’ benefits, and drawbacks is performed. The perceived average annual sq. m. rental value of the Sports club is calculated using the following formula (Table 4): (Nib + NIc) = NIa. If the formula is not fulfilled, this indicates that the perceived rental value of the Sports has not been evaluated adequately, and the cycle of approximation must be repeated until the formula is satisfied. During this stage, the perceived investment value of the Sports club that would make an investment project on average equal between the considered options was determined after 343 cycles. Thus, the Sports club needs a valence value of 0.3431 (initial is 0.0001) with a perceived average annual sq. m. rental of 139.43 (initial is 84) euro/m2 value to become the average equal investment alternative of the considered options. The difference between the perceived rental value and the rental value of the Sports club could be explained by the fact that emotional and affective states influence the project’s value (Table 4).

Table 4 Calculation of the perceived average annual sq. m. rental value of the Sports club.
Full size table

Circle map of the potential investment project and its context

For the apartment complex apartment complex project, a circle map of the potential investment project, its context, and the MAPS states was created. This apartment complex circle map analyzed 31 parameters (see Table 2 and Fig. 5), which illustrate the average opinion of potential investors and users of MAPS indicators and the set of averages of the building and its context indicators.

Fig. 5
Fig. 5
Full size image

Circle map of the apartment complex and its context and MAPS chart.

The apartment complex and its context data, as well as the MAPS states of potential investors and users presented in Table 2, were used to create the circle map, which is represented by the circular blocks in Fig. 5. The circle is divided into five sectors, each of which defines a set of criteria for the project under analysis. The building and economic criteria are divided into five smaller sectors, the social criteria into four, the environmental criteria into six and the emotional and affective criteria into 11. The entire quantity of sections is 31. The sections is separated into 10 cells. The cells reflect the emotional, affective, and sustainability ratings. For example, the average happiness of the passersby (potential investors and users) was 1.91 in the Ic location. This information is presented numerically in Table 2 and graphically in Fig. 5 by a light brown Sect. (1.91 points). Each colored block in the MAPS section characterizes the mean value of a specific emotion and affective attitude induced in potential investors and users in the Ic location. The higher the value, the larger the number of colored blocks.

The graphical mapping of the data visually highlighted the investment Ic project parameters such as the rental price and selling prices in addition to the liquidity of the property, as well as the emotional parameters such as happiness, valence, and affective parameters.

Making evidence-based recommendations

At this stage, for each investment project, the subsystem for making digital recommendations formulates suggestions to enhance specific criteria values and assess the effect of the updated values on the total project valuation.

The results of the evidence-based recommendations, based on steps 1–5, 9 and 10 of the INVAR approach97, are presented in Table 5. Each window in Table 5 describing the compared investment project consists of three parts: baseline data; a numerical suggestion demonstrating the percentage of enhancement in the value of the indicator relative to the optimal value of the chosen criterion; and a quantitative recommendation indicating the percentage of improvement in the project.

Table 5 Extract from the matrix of quantitative recommendations.
Full size table

Stage 9 of the INVAR approach97 determines the quantitative recommendation iij for the criterion xij indicating the possible enhancement percentage in the value of the criterion xij for it to match the highest value xi max of the criterion Xi. The following equation is used (see Table 5):

$$i_{ij} = \mid x_{ij} - x_{i \, max } \mid\: :x_{ij} \times \, 100\%$$
(1)

where iij is the quantitative recommendation for the criterion xij indicating the possible enhancement percentage in the value of the criterion xij for it to match the highest value xi max of the criterion Xi., when xi max is the best value of the criterion Xi of the alternatives under analysis.

Stage 10 of the INVAR approach97 determines the quantitative recommendation rij for the criterion xij indicating the possible enhancement percentage in the utility degree Nj of the option aj upon the achievement of xij = xi max. This means that rij indicates the potential percentage increase in the utility degree Nj of the option aj, given that the value of the criterion xij could reach the optimal value xi max of the criterion Xi. The following equation is used for the calculation:

$$r_{ij} = \, \left( {q_{i} \times x_{i \, max } } \right) \, : \, \left( {S_{ - j} + S_{ + j} } \right) \, \times \, 100\%$$
(2)

where rij is the quantitative recommendation for the criterion xij indicating the possible enhancement percentage of the utility degree Nj of the option aj upon the achievement of xij = xi max.

The calculated quantitative recommendations iij and rij for the criterion xij are then submitted in a matrix form (Table 5).

As an example, one of the criteria describing the projects is happiness. The gym (Ia), one of the investment projects analyzed, was assessed as the best option based on the happiness value (0.146) in Tables 2 and 3. If it were possible to improve the evaluation of the happiness for the shopping center (Ib) by 16.82% from the value obtained for 0.125 to the value obtained for the best project (Ia), 0.146, then the utility degree would increase by 0.96%. Similarly, if it were possible to improve the neighborhood quality for the apartment complex (Ic) by 33.33% from the value of 0.6 obtained to the best project (Ib) value of 0.8, then the utility degree would increase by 0.60% (Table 5).

The evidence-based recommendations presented at this stage show the percentage increase in the utility degree of the investment project that would be required to bring the indicator value up to the best indicator value.

Discussion and conclusions

The emergence of the latest technologies means that increasing amounts of big data are generated in green buildings. The World Green Building Council (WGBC) believes that green buildings help with the achievement of the UN Sustainable Development Goals and thus offer billions of people better living conditions. According to the WGBC, well-being has become an important consideration in many rating systems that score green buildings. Well-being is closely associated with happiness and satisfying quality of life, two parameters that can be measured by analyzing emotions. The availability of more detailed and real-time data on well-being and other aspects is, however, still inadequate. The collection of such data should preferably be low-cost.

Researchers from various fields have analyzed the impact of the built environment and its specific characteristics on our emotions and behaviors for more than five decades, and, more recently, have added our well-being to assessments. Much of the previous work has been stratified within individual disciplines. An increasing number of teams, however, now include humanities, arts, and social sciences along with science, technology, engineering, and medicine experts to bring industry and interdisciplinary knowledge to this field of research98. The rule that in purchasing decisions the share of logic is 20% and the remaining 80% is based on emotions99 is popular around the world.

Many researchers in the field of social sciences have sought to understand whether our psychological health is influenced by the built environment100. Yet they mostly used self-evaluated rating scales, descriptions, and questionnaires and thus analyzed this effect based on subjective indicators of psychological effect101,102. The environmental psychology term “behavior setting” describes the impact of the social environment (how others act and behave) and physical environment (place, time, and objects) on behavior103. Effective real estate investment depends on the information available and the influence of various factors, including emotional and affective states. Many researchers (Ekman104, Cherer and Ekman105) assume that good or bad emotions are the dominant driver of the most meaningful life decisions.

Other researchers (Zeile et al.106, Pykett and Cromby107, Mouratidis108) have examined the influence of urban spaces and their context on emotions by looking at a wide range of aspects. Their studies succeeded in many ways, but analysis of the biometric state, emotional state, valence, arousal, and affective states of potential buyers has not yet been integrated into any research of real estate investment and building sustainability ratings. The relationships between property and its context and its value, quality, and utility to people are important aspects to consider. This research analyzed five data layers comprising over 879 million remote data points accumulated over its course. In this sample of 879 million data points collected from November 2017 to January 1, 2023, the authors found a significant relationship between passer-by MAPS, pollution, and meteorological (MAPME) variables, and context. Based on this data, models for the arousal, valence, emotional (anger, disgust, happiness, neutrality, sadness, fear, surprise) and affective (confusion, boredom) states were developed (Table S1). MAPS, the property and its context details were used as a basis to create the holistic circle map of investments and MAPS states models.

The developed Multimodal Property Video Neuroanalytics (MOVE) collects big, real-time, and low-cost multimodal data and examines the property, its context, and potential investors’ MAPS. The developed MOVE integrates text, biometrics, audio and images sentiment analysis subsystems, regression and multi-criteria analysis methods. MOVE tackles the problem of four-modal sentiment analysis for the first time, demonstrating that it is a workable task that can gain from the combined use of text, biometrics, audio and images modalities.

Arellano et al.109 argue that emotions do not arise in a vacuum and depend on the context, which describes what happens around a person. The way children grow and develop depends on their environment in many important ways and their emotional development is affected by their school, family, culture, neighborhood, and peers110. When potential buyers react to ads with emotions, they tend to interpret the neutral details of a property through the lens of these emotions. If the emotions are negative, buyers often see these in negative light, whereas positive emotions often make them look positive. Emotional responses to property ads, rather than the actual ad content, often influence user intention to buy to a great extent. Multiple studies have pointed out that emotions play an important role in economic transactions and those changing feelings lead to changes in consumer goals111. Property ads should, therefore, include the purpose of property content, the property’s characteristics with an emphasis on added value by highlighting unique features, and high-quality photos and videos, taking into account the impact of emotional and affective states on investors’ attitudes toward the proposed acquisition of the property.

Ballas112 points out that socio-economic and geographical contextual aspects related to happiness, well-being, and quality of life represent an important factor, and emphasizes the impact of spatial and social inequalities, as well as social justice. In Western politics, the pursuit of happiness has long been seen as a primary political goal and social connectedness, health, and wealth are highlighted as key predictors of happiness in existing literature. Residents are happier in cities that provide easy access to cultural and leisure amenities along with convenient public transportation, that are affordable and offer a good environment for raising children113. Human-centered design is now very popular around the world. It merges the understanding of human needs with innovative solutions designed to meet these. The property context has also been confirmed to affect MAPS states; all these aspects are interrelated with the perceived investment value. The proposed methodology could be widely used by various stakeholders to analyze emotions in the built environment in the context of property investment, perceived investment value calculations, and calculating building sustainability ratings.

The Multimodal Property Video Neuroanalytics, MAPS, and the details of the properties in question and their context were used to present a holistic circle map of investment, incorporating the circumplex model of affect and evidence-based recommendations, and to calculate the perceived investment value of real estate. Research shows that properties and their context affect MAPS states; all these aspects are interrelated with the perceived investment value. The studies presented in Fig. 1, Table 1 and Section “Case study 1: Human diurnal rhythms, correlations and models in Vilnius city” provide evidence and theoretical arguments to bridge the gap between ambient public mood and specific market valuation.

The research is affected by restrictions typical of observational studies. The proposed Multimodal Property Video Neuroanalytics has some limitations. It is far from ideal and we can suggest a few ways in which it could be improved. One is to expand the Multimodal Property Video Neuroanalytics database by adding data on various green buildings and rating systems (LEED, BREEAM, CASBEE, Fitwel, German Sustainable Building Council, WELL, Green Star, Green Globes), events (festivals, concerts, conferences, exhibitions, workshops, seminars), monuments, and public spaces (courtyards, streets, squares, public archaeological sites). Another is to augment the system by adding brainwave and eye tracking sensors and a device to measure human electromagnetic fields. The mapping of emotions, affective states, biometric signals, and the context may make the urban maps more holistic and thus improve the accuracy of evidence-based recommendations on ways to boost the efficiency of the green buildings and context. Applying the best global practices, data on green buildings, public spaces activities and the Multimodal Property Video Neuroanalytics can improve the benefits of green buildings and public spaces.

It is our belief that MOVE can be used not only in simple context but also in more complex ones. This can facilitate launching new services and applications. The proposed MOVE can offer assistance to different stakeholders, including communities, mass media, universities, humanitarian organizations, non-governmental organizations, national and local governments, and regional institutions, and thus contribute to making communities more well-being, sustainable, and liveable. MOVE and MAPS models that exploit multimodal features can expand cross-disciplinary research further with possible benefits for technological advances in a wide range of areas, including economics, the environment, politics, culture, education, social life and healthcare.

Multimodal property video neuroanalytics

During the European Union’s Horizon 2020 (H2020) ROCK project, we developed the Video Neuroanalytics and connected sensors infrastructure in Vilnius (Lithuania) to perform research on passersby at ten locations in the city. The Video Neuroanalytics was further developed as part of the Horizon Europe META-MUSEUM and AFFECTS projects. The system upgraded during these projects was named Multimodal Property Video Neuroanalytics. Five layers of data were collected and then systematically evaluated for this study. The first three MAPS layers included (1) emotional states, arousal, and valence; (2) affective attitudes (confusion, interest, and boredom); and (3) physiological (heart and breathing rates, facial temperature, etc.) states. We collected and analyzed MAPS data in real time. The MAPS measurements were recorded every second. The fourth and fifth layers focused on pollution (based on data provided by the Environmental Protection Agency), as well as the presence or absence of magnetic storms and weather conditions (based on data provided by Vilnius Meteorology Station).

The Multimodal Property Video Neuroanalytics contains the following subsystems: an equipment subsystem, a relational database management system, relational databases, a model base management system, and an intelligent model base and user interface.

The equipment subsystem identifies the investor’s emotional, affective, and physiological states. The equipment subsystem (data layers 1–3) includes an emotion logic subsystem, Enobio EEG headset system, temperature screening (with an FLIR A35sc thermal camera), facial recognition software (Microsoft Azure Face API, FaceReader 7–9), heartbeat and respiration rate monitoring (with a Joybien BM201-VSD EVM Kit), a human vision components (HVC-P2) system (for detecting potential investors bodies and determining the level of happiness, neutrality, anger, surprise, and sadness, as well as the gender, age, blink rate, and face and gaze direction), a voice stress analyzer, and an X4M200 respiration sensor. During the study, the calibration of the sensors and measuring instruments of the equipment subsystem was performed. During the procedure, the readings of the sensors and measuring instruments were compared with a standard, a measuring instrument the accuracy of which is known and confirmed. This ensured the accuracy and reliability of the sensors and measuring instruments.

The structure of relational databases used in MOVE makes it easy to store, retrieve, modify, and delete data as various data processing operations are performed. These databases can establish relationships and links between MAPS and context data to help their users better understand the relationships between various MAPS and context data points and to gain specific insights. The Multimodal Property Video Neuroanalytics database contains the above five data layers. The relational database management system organizes data in a structured manner with rows and columns. It is intended to manage large amounts of structured data and can simultaneously support multiple users working at the same time.

The General Data Protection Regulation (GDPR), which came into force on May 25, 2018, applies in all Member States of the European Union112. Before launching our data gathering activities, we therefore, needed a data protection assessment for the equipment subsystem that enabled us to collect the data needed for the research.

The intelligent model base contains the following subsystems:

  • Multimodal analysis subsystem

  • Fusion subsystem

  • Property decision support subsystem

  • Subsystem for making digital recommendations

  • Subsystem for value analysis

  • Regression subsystem

  • Sentiment analysis subsystem

In our multimodal analysis subsystem, we consider a broad spectrum of modalities including biometrics, audio, images, text, and various possible combinations of the above modalities. Investment is a domain where analysis of more complex images is needed, and the context often plays a vital role in evoking people’s expressions, attitudes, feelings, opinions, and affective and emotional states. Many different combinations of two modalities can be used in MOVE, like text analysis integration with biometrics, audio, and images. Models can be either bimodal sentiment analysis models and use two modalities, or four-modal sentiment analysis models and use all four modalities. A multimodal analysis subsystem involves data identification, extraction, quantification, and multimodal synthesis, using heterogeneous sources and combining their different text, biometrics, statistics, audio, and image features. After the data are cleaned and appropriate selection is made using dimensionality reduction, the intelligent model base is used to extract audio, video, biometrical, and text features. For example, the voice analysis subsystem investigates the user’s vocal emotions applying the 38 parameters.

The fusion subsystem performs a synthesis of different modalities. This fusion means that multimodal data collected from the equipment subsystem are filtered to extract and combine required features, and then further analyzed to extract subjective information (opinions, attitudes), biometrical, affective, and emotional states. This fusion subsystem benefits each modality by allowing it to analyze its features applying the multimodal analysis subsystem. We used the late fusion strategy, which combines all of the modalities. The first step of this fusion subsystem is the independent processing and classification of each modality’s features. The second step is when the classification results are merged into the final conclusion and the prediction of opinions, attitudes, and affective and emotional states. The fusion subsystem uses the weighted averaging algorithm to synthesize data from different modalities (audio, image, etc.). Multimodal biometrics (such as face and voice) have different accuracy and reliability. Each participant of the experiment and the experts involved in this process also have their understanding of the significance of data from other modalities. One participant in the experiment, for instance, believes that her face conveys her emotional state much more accurately than her voice. Taking these three aspects into account, the expert, together with the participant, determines the weight of specific modalities and their measured variables using the weighted averaging algorithm. The standardized β coefficients calculated in the regression models are then multiplied by the weight of particular measured variables (Table S1). In our specific case, the weight of all measured variables was equal to 1.

The aim of a property decision support subsystem is to assist property professionals in making multiple criteria decisions about property acquisition, enhancement, and disposal by offering data, models, and an interface to minimize uncertainty and enhance evidence-based decision quality. It combines databases, models, and interfaces to assess real estate options, thereby improving efficiency and reducing decision-making duration in property investment and management. The database contains organized quantitative and qualitative details regarding market data, properties, building attributes, context, and economic factors. The model base comprises two models, which are based on the multiple criteria analysis methods COPRAS114 and INVAR97, to help assess real estate options, examine different situations (developing, purchasing, renovation, etc.), and evaluate their possible effects on portfolios, which can be a labor-intensive and monotonous job for investors. The COPRAS method, for instance, was cited in 733 scientific articles in the Google Scholar database. The User Interface offers an engaging platform for brokers and investors to enter and retrieve data, and perform multiple criteria analysis of alternatives. The investor employs a property decision support subsystem (DSS). A property DSS is able to assemble a many details and information from necessary, relevant data for analyzing real estate investment and building sustainability ratings and making decisions. Real estate investment and building sustainability ratings can be simulated by a property DSS in groups or individually, for adequate visualization. The database of a property DSS can store and develop the structured and unstructured data that real estate investment and building sustainability ratings experts accumulate. These accumulated investment and ratings data can be used to examine investment projects, building sustainability ratings, and contexts.

Application of the subsystem for value analysis and subsystem for making evidence-based digital recommendations enables investors to create their own DSS and recommendation systems, as well as to calculate market, customer-perceived, investment, hedonistic, emotional, and utilitarian values and provide evidence-based digital recommendations for improving projects. These subsystems use multiple criteria analysis based on neuro-decision matrices. Such multiple criteria analysis has been applied in many studies115,116,117,118 where emotional, affective, and physiological data from passersby were analyzed alongside the traditional quantitative and qualitative analysis indicators of the built environment. Thereby, the Multimodal Property Video Neuroanalytics assists stakeholders in better holistically comprehending real estate investment and building sustainability ratings.

Systematic reviews of observational studies are frequently conducted worldwide. In this observational, correlational study, the variables of passer-by MAPS, pollution, and meteorological (MAPME), context (Supplementary materials 3), and perceived rental value were observed and measured to determine whether there is a relationship between them. This is a form of observational and regression analysis research specifically designed to examine the relationship between the MAPME variables, context, and perceived rental value.

The regression subsystem develops MAPS states regression models. The regression subsystem analyzes the connection between dependent valence, arousal, happiness, interest metrics, property, and its context-independent metrics. It is also applied for forecasting customer-perceived value. With the help of the regression subsystem, MAPS states models were developed using stepwise regression. We used our research data as the basis to create regression equations for the valence, arousal, emotional (angry, disgusted, happy, neutral, sad, scared, surprised) and affective (confusion, boredom) states (Table S1). Stepwise regression is a special case of hierarchical regression in which statistical algorithms determine what predictors end up in your model. Predictors may be reduced by using stepwise regression119. Stepwise regression algorithm is an automatic procedure for statistical model selection in cases where there is a large number of potential explanatory variables, and no underlying theory on which to base the model selection120. The procedure is used primarily in regression analysis, though the basic approach is applicable in many forms of model selection121. The diurnal valence, arousal, emotional (angry, disgusted, happy, neutral, sad, scared, surprised) and affective (confusion, boredom) states regression models were developed and assessed applying R2, the beta and standardized beta coefficients β, the p-value, and the elasticity coefficient.

The sentiment analysis subsystem routinely diagnoses, classifies, and assesses the emotional trends expressed in the speaking text and provides a transcription with emotional metrics.