Memory as an Autonomous System

Figure 1 schematically illustrates the entire structure of MHP/RT. From the viewpoint of synchronization with the ever-changing environment, MHP/RT is divided in two different systems. The first one resides in the Synchronous Band that operates in a way that is tightly coupled with the ever-changing environment, including Perceptual Information Processing System, Conscious Information Processing System, Autonomous Automatic Behavior Control Processing System, and Behavioral Action Processing System, arranged in the thick yellow arrow in the center of the figure. The second one resides in the Asynchronous Band that works quasi-independently with the environment and serves for regulating the processes in the Synchronous Band. Memory Processing and Body Condition Monitoring constitute the Asynchronous Band.

All the functional modules shown as round-corner-rectangles are autonomous systems. In MHP/RT Memory System is not subordinate to the other systems but has its autonomy in reacting to the input from the Perceptual Information Processing autonomous system. There is no arrow from conscious and unconscious processes to Memory Processing but there are arrows to those processes to transfer information by means of resonance reaction.

Figure 1. Six autonomous systems in MHP/RT, arranged in Synchronous Band and Asynchronous Band.

Memory reaction under real time constraints

Memory processes include storage of information and usage of stored information. In order to carry out real time simulation of human action selections in daily life, memory processes play very important role. As Allen Newell described in his seminal book, "Unified Theories of Cognition", human beings operate differently according to the characteristic times that the environment they are in (Table 1).

Table 1. Newell's Time Sale of Human Action.

---

Scale (sec)	Time Units	System	World (Theory)
107	months
106	weeks		Social Band
105	days
104	hours	Task
103	10 min	Task	Rational Band
102	minutes	Task
101	10 sec	Unit Task
100	1 sec	Operations	Cognitive Band
10-1	100 ms	Deliberate Act
10-2	10 ms	Neural Circuit
10-3	1 ms	Neuron	Biological Band
10-4	100 μs	Organelle

---

In parallel with Newell's, we depict in Figure 2 how the memory system should react to the environment depending on the degree of time constraints that the human–environment system, or organic self-consistent field, should impose on itself. When real time constraints are strong, slow memory processes that use long-term memory should not be able to participate in the whole processing. In other words, only the unconscious side of the Two Minds system should work. On the contrary, when there are little real time constraints, both systems, consciousness and unconsciousness systems work collaboratively in some cases and independently in other cases.

Figure 2. Memory reaction under real time constraints.

Top

Two views of memory structure

There are two views of memory structure. One is "memory as a conceptual construct" and the other is "memory as a nervous system." Integration of these two views should provide a deeper understanding of human memory. The goal of this chapter is to reconsider MHP/RT's autonomous memory system by mapping it on the real interconnections between the cranial nerves and the spinal nerves. This results in a topological representation of the distributed memory system.

> Memory as a conceptual construct

In the diagram of Two Minds depicted by Kahneman (2003; A perspective on judgment and choice, American Psychologist 58 (9) (2003) 697 - 720), the memory module is placed independently from System 1 and System 2. MHP/RT extends Two Minds by specifying the role of memory when System 1 and System 2 work jointly. This is critical because the contents stored in memory determine what System 1 and System 2 would do in the ever-changing environment, and we are interested in interactions between the two systems with different characteristic times, i.e., very fast System 1 and very slow System 2.

Figure 1. Structure of memory.

In MHP/RT, the human memory structure is modeled conceptually as shown by Figure 1: MHP/RT assumes that memory is organized by "Multi-Dimensional Frame (MD-Frame)" for storing information. MD-frame is a conceptual ex- tension of Minsky's frame. It is a primitive cognitive unit that conveys information that can be manipulated by brain under various constraints, similar to the concept in the ICM (Idealized Cognitive Model) theory by G. Lakoff, the schema theory by D. E. Rumelhart,, and so on. There are a number of MD-frame in MHP/RT.

> Memory as a nervous system

On the other hand, in reality, memory is a structured record of the activities of the nervous system to internal and external stimuli. The nervous system consists of three sub-systems (L. W. Swanson, Brain Architecture, Oxford University Press, 2011).

1. Sensory nervous system, that is responsible for processing sensory information,
2. Somatic nervous system, that is associated with the voluntary control of body movements via skeletal muscles, and
3. Interneurons nervous system, that connect sensory nervous system and somatic nervous system.

Each nervous system is functionally different from each other in terms of its input-output structure, and therefore each is associated with its own memory structure, that develops and organizes differently from the other memory structures.

Each memory structure interconnects hierarchically as the nervous system develops, topologically forming an evolving cyclic network structure with the somatic nervous system being the core of the network.

Memory formation and activation

Since one's birth, one continuously carries out physical behavior without any pause. This reflects on the construction of the respective memories: each constitutes a continuous cyclic network that reflects day-by-day cyclic events. Perception plays an important role for the development of the cyclic memory network. The memories are cross-connected each other and perceptual objects reside at the center of the cross connections. The perceptual system continuously monitors changes in the environment, and starting from the perceptual objects that are created while processing the environment, it fires the cyclic memory network. The firing continues and spreads in the network, i.e., chain-firing, as long as the perceptual objects last in the environment. The activated region of the network finally obtained as the result of the chain-firing is "the accessible memory region" at the moment. What this process actually does is to preprocess the situation for the future by activating related memory regions beforehand.

Figure 2. Chain-firing triggered by perceptual stimuli in a cross-networked memory structure.

Figure 2 illustrates schematically the process of chain-firing triggered by perceptual objects. The accessible memory region, depicted in the red oval in the figure, corresponds to the region where the activation levels are kept above certain threshold values necessary for maintaining a chain-firing by using primarily perceptual stimuli related to the current behavioral purpose. The origins of the chain-firing are continuous input from the current environment. In the situation with severe realtime constraints, the activated region would be used with a high priority.

In summary, specific input stimuli, which could be internal or external, selectively activate part of memory, and it then triggers activation of another part of memory. This continues as long as input stimuli exist. Since the sensory nervous system reacts to the rapidly-changing external environment with shorter reaction times, the activated part of memory tends to be related with the current state of the environment.

Multidimensional frame as a distributed memory system

In MHP/RT, object cognition occurs as follows:

• collecting information from the environment via perceptual sensors,
• integrating and segmenting the collected information, centering on visually collected objects, and
• continuing these processes until the necessary objects to live in the environment are obtained.

These objects are then used independently in Systems 1 and System 2 of Two Minds, and memorized after integrating related entities associated with each system. Due to the limitation of the brain's processing capability, the range of integration is limited; therefore, System 1 memory and System 2 memory should differ. However, they could share objects originating from perceptual sensors. Thus, when objects that are the result of the just-fiished integration and segmentation process are processed in the next cycle, representation of the objects may serve as the common elements to combine the System 1 memory and the System 2 memory to form an inter-system memory. We call this memory the Multi-Dimensional (MD) -frame.

There are five kinds of MD-frame in MHP/RT.

Figure 3. MHP/RT and the distributed memory system.

Figure 3 illustrates how each MD-frame is created as the result of working of autonomous processes in MHP/RT and how MD-frames are mutually interrelated. This essentially details the process "Memorization of Behavioral Actions and their Results" shown by the dotted oval in the diagram of MHP/RT by considering neuronal activities that actually happen. The basic idea is that each autonomous system has its own memory.

• PMD (Perceptual Multi-Dimensional) -frame constitutes perceptual memory as a relational matrix structure. It collects information from external objects followed by separating it into a variety of perceptual information, and re-collects the same information in the other situations, accumulating the information from the objects via a variety of different processes. PMD-frame incrementally grows as it creates memory from the input information and matches it against the past memory in parallel.
• MMD (Motion Multi-Dimensional) -frame constitutes behavioral memory as a matrix structure. The behavioral action processing starts when unconscious autonomous behavior shows after one's birth. It gathers a variety of perceptual information as well to connect muscles with nerves using spinals as a reflection point. In accordance with one's physical growth, it widens the range of activities the behavioral action processing can cover autonomously.
• BMD (Behavior Multi-Dimensional) -frame is the memory structure associated with the autonomous automatic behavior control processing. It combines a set of MMD-frames into a manipulable unit.
• RMD (Relation Multi-Dimensional) -frame is the memory structure associated with the conscious information processing. It combines a set of BMD-frames into a manipulable unit. The role BMD-frames play for RMD-frame is equivalent to the role MMD-frames play for BMD-frame.
• WMD (Word Multi-Dimensional) -frame is the memory structure for language. It is constructed on a very simple one-dimensional array.

Functional flow structure, layered structure, and evolving cyclic network structure

Figure 4 illustrates how the MD-frames are interrelated by introducing three structures representing different view points. These three structures enable pipelining the processes. This cyclic connection is critical to understand the relationship between behavior and memory, which is not appropriately dealt with in the neuroscience researches.

Figure 4. Functional flow structure, layered structure, and evolving cyclic network structure.

> Functional flow structure

The functional flow structure describes memory activation paths starting from PMD-frames to WMD-frame, RMD-frame, or BMD-frame to MMD-frame, which describes how perception triggers motions. Notice that MMD-frame is the terminal MD-frame and therefore the paths from WMD-frame to RMD-frame, and RMD-frame to BMD-frame exist.

> Layered structure,

The layered structure consists of the following three layers:

1. PMD-frame -- WMD-frame layer is the top layer controlled by words. It consists of simple one-dimensional array of symbols, logically constructed language, grammars that specify language use, etc.
2. PMD-frame -- RMD-frame layer is the middle layer that resides on the behavioral eco-network for the individual. In this layer, one acquires the meaning of behavior in the social ecology.
3. PMD-frame -- BMD/MMD-frame layer is the bottom layer that creates a behavioral eco-network for the individual. This is a cyclic network starting from PMD-frame towards MMD-frame, and returning to PMD-frame.

> Evolving cyclic network structure

The evolving cyclic network structure depicted by a blue spiral refers to the fact that respective autonomous systems generate distributed memories for their use, and the memories are cyclically related and in effect topological.

Top

Memory as a Mechanism to Establish Synchronization

It is important to note that memory serves as a mechanism to establish synchronization. A rough sketch concerning how synchronization between the external world and the internal world could be done can be drawn by combining MHP/RT and Distributed Memory System. Here is how a PCM cycle is initiated and continues.

1. When sensory neurons are activated by external stimuli, Perceptual Information Processing fires relevant portions of memory followed by firing of connected networks in the current memory structure, and
2. The activated portions of memory are available for System 2 and System 1 via resonance reaction for some amount of time.
3. Then, finally the output of System 1 is input to Behavioral Action Processing to carry out behavior by sending signals to the associated motor neurons to act in the real world.
4. Motor neurons move body parts, which lead to changes in the state of the self, during which the external world may have changed — This is the time when a certain EVENT happens.

The movement of one’s body part will initiate the next cycle of PCM processes. What is going on in the external world and what the self is behaving are internalized through perception, and the perceived world goes through a PCM process by activating relevant portions of memory and utilizing it via resonance mechanism. In this way, memory processes play an important role to bridge the gap between the perceived external world and the internal world of PCM cycles to have them go coherently, i.e., keeping them synchronized.

The second and fourth processes are critical to understand how synchronization is carried out as follows:

• At a particular time before the event, one engages in conscious processes and unconscious processes concerning the event to happen in the future.
• At a particular time after the event, one engages in conscious processes and unconscious processes concerning the event that has already happened in the past.

What one can do before and after the event is strongly constrained by the Newell’s time scale of human action. System 2 carries out the processes in the upper bands of Table 1, whereas System 1 does in the lower bands of Table 1.

Four Processing Modes (Relative to Event)

When focusing on a certain event, MHP/RT works in one of four different ways: Two are before an event to happen in the future in which MHP/RT uses memory. And the other two are after the event in which it modifies memory.

• In System 2 Before Mode, MHP/RT consciously uses memory before the event.
• In System 1 Before Mode, it unconsciously uses memory just before the event for automatic preparation for the future event.
• In System 1 After Mode, it unconsciously tunes the current network connections related to the past event for better performance for the same event in the future.
• In System 2 After Mode, it consciously reflects on the past event resulting in structural changes in memory.

Table 1 summarizes the features of the Four Processing Modes.

Table 1. Four Processing Modes.

---

	System 2 (Conscious Processes)		System 1 (Unconscious Processes)
	Before	After	Before	After
Time Constraints	none or weak	exist	none or weak	exist
Network Structure	feedback	feedback	feedforward + feedback	feedforward + feedback
Processing	main serial conscious process + subsidiary parallel process	main serial conscious process + subsidiary parallel process	simple parallel process	simple parallel process
Newell’s Time Scale	Rational / Social	Rational / Social	Biological / Cognitive	Biological / Cognitive

---

Figure 1 illustrates the Four Processing Modes along the time dimension expanding before and after the event. It is useful to consider that at each moment along the time dimension human behaves in one of the four processing modes and switches among them depending on the internal and external states.

• T - β is the time when starting to consciously consider the future event which happens at unknown future time T.
• T - β' is the time when switching to System 1.
• T + α' is the time when terminating System 1 processing.
• T + α is the time when finishing System 2 processing.

Figure 1. Illustration of the working of the four processing modes.

In System 2 Before Mode, MHP/RT uses memory of words and relations for consciously preparing for what would happen in the future.

Figure 2. System 2 BEFORE Mode.

In System 1 Before Mode, it unconsciously coordinate motor activities to the interacting environment. This mode uses the memory of perception, behavior and motor movement.

Figure 3. System 1 BEFORE Mode.

In System 1 After Mode, MHP/RT unconsciously tunes the connections between sensory inputs and motor outputs for better performance for the same event in the future. This mode updates the connections at the bottom layer of the topological representation of MD-Frames, which is Perception, Behavior and Motor links.

Figure 4. System 1 AFTER Mode.

In System 2 After Mode, System 2 consciously reflects on the past event by consciously recognizing what has happened, and then, modifies memory concerning the event. This mode modifies the connections at the middle layer of the topological representation of MD-Frames, which is Perception and Rule links.

Figure 5. System 2 AFTER Mode.

Nature of Consciously Retrievable Experience

The event that occurs at T is represented from the users perspective as follows:
Any event can be viewed from a user as an event that has happened in the time range of [T - β, T + α] accompanied with the portions of MD memory frames that have participated in the processes in the time range of [T - β, T + α].

The event that occurs at T is consciously retrievable and has the following features:

• It is important to notice the fact that consciousness concerning the event comes to play in System 2 After Mode at T + α'. This implies that consciousness lags at least by the amount of time, α' behind the real world. Therefore, the user is consciously blind during the period of [T - β', T + α']. But would integrate consciously the blind period into conscious activities during the time range of [T-β, T + α].
• Therefore, consciously retrievable experience has to be considered as memory structure concerning the event at T in the extended time range of [T - β, T + α] with consciously inaccessible but integrated memory region corresponding to the consciously blind time range of [T - β' , T + α'].

Weak Synchronization

External system event at T is mapped internally as a consciously retrievable event. More specifically, a user’s activity related with an event has to be considered from the four processing modes, which ranges relatively long time before and after the actual time the event happens. Therefore, “synchronization” has to be considered alternatively as the phenomena as follows. A user’s activities during the time range of [T - β,T + α] are linked with the specific recognizable system event at time T. And the activities are carried out in either of the four processing modes. When this is satisfied, the event is considered synchronized with a user’s activities, which is called weak synchronization.

Figure 6. How MHP/RT Works under Weak Synchronization.

Figure 6 schematically shows how MHP/RT works under weak synchronization during the time range of [T - β, T + α].

• During [T - β, T - β'], activities at upper bands, social, rational and upper cognitive bands are carried out consciously,
• The conscious activities are taken over to unconsciously performed activities, which are carried out during the period of [T - β', T], the time when the event happens.
• It follows unconscious tuning activities during the period of [T, T + α'].
• Finally, these activities are taken over to the activities carried out during the period of [T + α' T + α] for conscious reflection.

Now, we need a mechanism for establishing chains between activities.

Firstly, we think about what are combined. An O-PDP system is composed of autonomous elements in a band-structure The bands are: Biological, Cognitive, Rational, and Social Bands as suggested by Newell. Activities carried out in the respective bands are combined each other to form a function, coherently integrated for the system event at T.

Then, how the activities are combined? The combined entity indirectly reflects the circularity of the existing environment and fluctuations inherent in the environment, which is called autopoiesis. Circularity connotes relativization of time: no distinction between “after” and “before”, fluctuations indicate that the resulting system is a complex system.

The event appears recursively in the PCM processes, and is likely to extend the relevant time range longer, i.e., beta and alpha would become larger.

• The larger β becomes, the farther the person can foresee.
• The larger α becomes, the wider the person can elaborate on the event.

These should make the person smarter in living.

Now we consider what results from weak synchronization. Firstly, various combinations of chains between parallel processes can occur, causing path proliferation of the network. When the recall rate of specific paths becomes higher, proliferation along these paths are suppressed to centralize the activations on these paths thereafter. It makes possible for the O-PDP system to survive in the cyclic and fluctuating environment. Each element of the O-PDP system plays a certain role in achieving the overall goal of the whole O-PDP system. However, its role is not determined from the beginning but it exists only as the result of each element’s own efforts to survive since it started its activity.

Top

Continuous cyclic loop of perception and movement

What we observe as each individual’s physical behavior is the results of a multiple processing with a PDP system, not with a single unified system. This PDP system is organized evolution- ally, and realized as a neural network system, including the brain, the spinal nerves, and the peripheral nerves.

According to Antonio Damasio ("Feeling of What Happens: Body and Emotion in the Making of Consciousness", 1999), a vertebrate animal develops its neural network system in the following way. It starts with the development of the paired structure consisting of the sense of touch and reflexive movements associated with it. Then the sense of smell and the sense of taste, and finally, the sense of seeing and the sense of hearing develop their associations with reflexive movements. From the beginning, the perceptual stimuli from the five senses form a paired structure with their associated reflexive movements. In addition, the association tends to become bidirectional for the purpose of establishing selective sensing, which is a paired structure with feedback between perception and movement. For example, the sense of hearing and the sense of vocalization establish a feedback loop between them immediately after one acquires the function of voicing.

In summary, the neural network system forms at first the autonomic nervous system of respective autonomous organs as a genetic fundamental structure, then crosses it with the somatic nervous system that controls reflexive movements associated with the perceptual stimuli from the five senses, and develops the feedback loops with a system of interneurons that connect these systems. Figure 1 depicts this loop schematically.

Figure 1. Continuous cyclic loop of perception and movement.

Formation process

As Larry W. Swanson explains in his book "Brain Architecture (Oxford University Press, 2011)", cross-networks of neurons in the brain develop in a systematic way to show three-layered structure of interneurons system (Figure 2). Interneurons intervene the sensory nervous system that is responsible for processing sensory information and the somatic nervous system that is associated with the voluntary control of body movements via skeletal muscles to form complex paired structure of perception and motion. They consist of direct feedforward connections from perception to motion, and more complex connections with feedback loops using the interneurons to form three distinguishable layers.

Figure 2. Development of the sensory nervous system and the somatic nervous system, and interneurons connecting them with action selection process.

The following explains the developmental paths of the neural networks as human beings grow as the function of their ages.

> Early stage: 0 ~ 6 years of age

In 0 ~ 6 years of age, feedforward loops are the dominant control mechanism and they establish fundamental relationships between the layers by means of “uplink.” In the first half of this period, 0 ~ 3 years of age, human beings establish inter-connections between Layer-1 and Layer-2 as integrated movements of bodily actions on the basis of the relationships between the input from the perceptual system and the output expressed as reflexive movements, for example, simple utterances. In the latter half of this period, 4 ~ 6 years of age, human beings acquire the skill of behaving in relation with the other persons and the methods for conversing with others such as explanation formation via simple syntax.

Up to 3 years old:Initial formation of the basis of sensory and motor organs:

Formation of the basis of feedback neural network is characterized by the following items:

• Development of the basic ecosystem of vertebrates before activation of conscious activity -- Corresponding to the stage of brain development of stickleback fish.
• Autonomous development of both perceptual and motor functions.
• Generation of cognitive memory in the part that mediates perception and movement.

During this period, sensory and motor neurons make a rapid autonomous development. Interneurons connect them by subordinate-intervention, resulting in feedforward control.

Figure 3. Development of PIM: Up to 3 years old.

Stochastic production-like action selection is performed reflecting the effectiveness confirmed by the experience accumulated so far.

Figure 4. Development of memory: Up to 3 years old.

During 6 ~ 7 years old: Activation of consciousness mechanism

Acquisition of utterances and combined motor movements become possible by realizing feedback involvement of consciousness mechanism to body mechanism.

Sharing of perception is required because It is necessary to share the perception of the phenomenon between System 1 and System 2 to implement feedback control.

• Direct Perception Sharing: Cognition via sensory nerve activity related to perception (tactile) resulting from physical activity (internal cognition).
• Indirect Perceptual Sharing: The visual cognition of a phenomenon and the perception by non-visual modalities that occur in relation to the phenomenon occur simultaneously to share their perceptual cognition (Perceptual Synchrony).

During this period, sensory and motor neurons make a fast autonomous development. Interneurons connect them by memory-mediated intervention, resulting in feedback control.

Figure 5. Development of PIM: : During 6 ~ 7 years old.

In a multi-dimensional memory frame that acts as a naturally formed perceptron, action selection is made by branching paths in the process of following the firing chain of elements in the MD frames, that reflects accumulated experience to work as feedback.

Figure 6. Development of memory: During 6 ~ 7 years old.

> Middle stage: 7 ~ 13 ... 17 years of age

Later, in 7 ~ 12 years of age, human beings acquire the skill of logical thinking by means of the first order logic by using letters or symbols and that of cooperation with the other persons. These activities facilitate the development of interconnections among the three layers, resulting in very complex networks. The key is the existence of symbols that intervene various connections between input and output.

Lastly, in 13 ~ 18 years of age, feedback loops come into play, which are used to form language processing circuits in a single layer, Layer 3, by means of the learning mechanism such as the back propagation.

In 13 ~ 18 years of age, the interconnections of the neural networks evolve among the three layers. In this period, the ability of logical writing by using an ordinary language affects significantly the evolving process. Without language, structural recognition is formed dominantly via visual information. On the other hand, when accompanied with language, it makes possible to represent the visual information in a highly logical way, the vision-based structural recognition is significantly augmented to become a structure that can be dealt with a language-based logic system.

Completion of parallel distributed activities: 7 ~ 13 ... 17 years of age

Structural development precedes procedural development. In time, both interact with each other to achieve unique development in parallel.

1. Structural development in PMD (mainly visual perception) → BMD → MMD.
2. Procedural development in RMD: starting from spoken language to more abstract structural representations

The combination of the above two elements formed for each person becomes the action route of that person.

As human beings stand up, the following things have happened:

• The flow of information flow becomes stronger at the junction between sensory related nerves and motor spinal nerves. This creates a new area in the brain's neural circuit that causes information stagnation like so-called delta.
• Human beings develop their pronunciation function (motion), acquire voice, and link various sounds with auditory (perception) processing. And, by making effective use of the vacant area of the brain, the function of communication (brain) which has already been used for speech can be expanded and developed into a language.
• The difference in the function of the limbs and the difference in the movement between the left and right become clear, and the action becomes complicated. The brain develops processes individually and integratively to their actions.

Up to 13 years old, parallel distributed activities are almost complete. Sensory and motor neurons make a stable autonomous development. Interneurons connect them by memory-mediated proactive-intervention, resulting in wider feedback controls.

Figure 7. Development of PIM: Up to 13 years old.

Up to 13 years old, parallel distributed activities are complete. Sensory and motor neurons make a settled autonomous development. Interneurons connect them by memory-mediated autonomous-intervention, resulting in even wider feedback control.

Figure 8. Development of PIM: Up to 17 years old.

> Final stage: 18 ~ years of age

Finally, in 18 ~ years of age, feedback loops become dominant, which make possible to form a compound language processing circuits by means of the learning mechanism such as the back propagation mechanism.

Human behavior development and language during this final stage is achieved through the following steps:

1. Acquisition of a technique (grammar) for mapping perceptual information related to actions to be executed in 4-dimensional space-time on 1-dimensional language, and making it meme (MMD, BMD, and PMD to WMD mapping).
2. This made it possible to reverse the flow in (1), i.e., it has become possible to construct a correlation between a simple neural connection layer and a basic neural connection layer based on complex multidimensional perception (via RMD).
3. By these, human beings succeeded in forming the relation between the hierarchy easily in a short time from both ends of the hierarchy.
4. In addition, since language promotes the establishment of common cognition among groups, it enables communication between groups and generations. This paved the way for efficient external cognition and higher adaptive behavior.

Figure 9. WMD is uniquely extended based on partial mapping of RMD.

> Summary: Stages of development

1. Up to 3 years old: Perceptual and motor organs acquire basic performance through experience so that their abilities adapt to the environment
2. Until about 6 ~ 7: Acquire an individual's ability to carry out their activities as individuals and as members of a group without a shortage.
3. From about 13 ~ 14: Enhance their ability to adapt to their role in social (group) activities.

These changes are caused by the shift from 2-Process to 4-Process:

1. 2-Process: Formation of neural circuit to execute normal processing efficiently by imitation
2. 4-Process: Dealing with abnormal situations where danger avoidance is required by the formation of new behavioral pathways that can be configured by existing normal-state processing

Neoteny

Neoteny is the delaying or slowing of the physiological (or somatic) development of an organism, typically an animal. It applies to development of perception, interneuron, and motor mechanisms in the individual environments.

> Development of perception, interneuron, and motor mechanisms in the individual environments

Figure 2 explains how neoteny occurs in an O-PDP system.

1. Components of O-PDP develop autonomously.
2. Mutations also occur for each component.
3. Development of O-PDP occurs on the basis of the body plan that regulates the flow of inter-component chains and the order of synchronization between components.

Figure 10. Hierarchical evolution by successive re-synchronization among perceptual, intervening, and motor mechanisms (a detailed explanation of Figure 2).

Therefore, when a change occurs in one of the components, the influence extends to the entire component and the entire structure in various forms.

When a certain autonomous mechanism develops by a dominant mutation, the synchronization mode between the autonomous mechanisms is readjusted, and mutations of other autonomous mechanisms are promoted.

> Change of active bands – followed by route generation, selection, and extinction

Changes of active bands occur due to environmental changes or due to ecological changes of the competitive group. Changes are recognized when the following events happen:

• Activity constraints become stronger.
• Noticing being freed from the activity constraints in continuous stable activity by accidental changes of own behavior or changes in perceptual information.

Changes of active bands cause changes in the route of the ecological network, but there is no fixed way of changing the route, so long as it is persistent. After that, route generation, selection and extinction are repeated, and overall optimization is promoted from the viewpoint of overall circulation efficiency.

> Development through selecting efficient splicing routes

The development of the O-PDP system mechanism follows discrete paths by splicing rather than continuous. Conditions for selecting efficient splicing routes are as follows:

• The problem or goal is clearly recognized.
• Noticing that multiple routes leading to the goal are available.
• Either SYSTEM 1 or SYSTEM 2 takes the initiative at the appropriate time to search for efficient routes to the goal.
• A variety of solutions are possible because they depend on the balance of SYSTEM 1 and SYSTEM 2 at that time.

> Neoteny and splicing

Non-linear development in neoteny interacts with non-linear development in splicing to create behavioral traces in the real environment, with the following features:

• Manifestation of dual processes, i.e., conscious processes, SYSTEM 2, and unconscious processes, SYSTEM 1.
• Exponential differences in characteristic times of active bands.
• SYSTEM 1-1 and SYSTEM 1-2 are related to each other and perform functionally distributed parallel activities, whereas SYSTEM 2 forms serial and unified action paths.

Top

Summary

This paper presents evidence that even if two individuals exhibit similar reactions caused by their autonomous nervous systems towards the same external stimuli, one may form a memory that can be recalled consciously, whereas the other may form a memory that cannot be processed consciously. We had fifteen monitors watch films, and we recorded their heart rates (HRs). Two monitors exhibited similar autonomous nervous system reactions, measured by HR, to the same films. We conducted interviews and compared their episodic memory about the films that both watched. One monitor was able to verbalize in detail her behavior, whereas the other could recall little of what he had watched. We suggest that this phenomenon can be explained by assuming that for the two monitors System 1 of Two Minds worked similarly to external stimuli, whereas System 2 worked differently for each monitor, affecting the formation of episodic memory in terms of conscious recall-ability.

Introduction

The retrospective interview is an efficient method for clarify-ing people’s decision-making process. We developed a retrospective interview-based study methodology, called Cognitive Chrono-Ethnography (CCE) (Kitajima, Nakajima, & Toyota, 2010). We then conducted a series of studies on the behavior of tourists at hot spring resorts (Kitajima, Tahira, & Takahashi, 2010; Kitajima, Tahira, Takahashi, & Midorikawa, in press) and of spectators of professional baseball games (Kitajima, Nakajima, & Toyota, 2010) in an effort to find an effective methodology for understanding the relationship between an individual’s experience and observed behavior at the moment, and chronological development. CCE is based on the assumption that human beings’ decision-making should be considered the result of the workings of Two Minds (Evans & Frankish, 2009) and human beings’ memory should be considered an autonomous system (Marcus, 2008). This approach is modeled after the Model Human Processor with Real-Time Constraints (MHP/RT) developed by Toyota and Kitajima (2010).

The retrospective-interview-based methodology uses consciously recallable episodic memory to understand monitors’ observed behavior. However, when one is experiencing an event, his or her Two Minds are working. Since System 1 (automatic processes) and System 2 (controlled system) of the Two Minds work in parallel and differ from person to person, the result may be the formation of individually different episodic memory for the same event. Therefore, apparent similarity in observed behaviors does not always guarantee that similar episodic memory has been formed. Formation of episodic memory should be considered a process other than experiencing external stimuli in real time to react immediately.

This paper describes the investigation of the relationship between monitors’ experience and their episodic memory when watching short films for the first time in a movie theater. The movie theater provides the ideal environment for this study because all the information comes from a big screen and a set of loud speakers in a synchronized way so as to occupy the monitors’ senses of sight and hearing completely. Thus, they are forced to concentrate on watching the films.

In this paper, we start by describing the study field, which is the Sapporo Short Film Festival (SSF), present an outline of the field study, and discuss the results to show evidence of similar experiential behavior accompanied with totally different episodic memory formation processes. We then introduce MHP/RT briefly and explain how MHP/RT explains this phenomenon.

Observation and Retrospective Interview

This study consisted of observation at the festival and follow-up retrospective interviews. The field observation was conducted at SSF2009, and the interviews were conducted a few weeks after the festival.

> SSF2009

The SSF started in 2006 and is organized by the City of Sapporo, Hokkaido, Japan, and the SSF executive committee. SSF is held annually in the autumn for one week in downtown Sapporo. This study involved the observation of study monitors while they watched short films at SSF2009. SSF2009 was held October 14 through 18, 2009, and showed 103 short films organized into fourteen 90-minute programs.

> Outline of the Study

A CCE study was conducted to observe monitors’ behavior at SSF. Study monitors participated in SSF in a manner that was typical of film viewers. Their activities were recorded during two 90-minute programs. Each monitor was required to attend two 90-minute retrospective interview sessions in the following month.

Recruiting Monitors. We conducted a Web survey and recruited fifteen monitors who had different attitudes towards viewing films. A Web questionnaire survey to screen potential monitors was distributed from August 12 through August 19, 2009, to people who lived in the Hokkaido area. A total of 232 respondents provided valid answers and demonstrated a willingness to participate in the study. By analyzing their responses, we identified five groups with different characteristics (Table 1). We finally chose 15 study monitors (six males and nine females, 21 to 56 years old).

Table 1. Characteristics of Study Monitors.

---

Group	Experience of Film Festival	Explanation	Number of Monitors
G1	yes	Have had an interest in short films for a long time. Have recently joined film festivals.	4
G2	yes	Didn't have interest in short films before. Began to like them after first joining a short film festival.	1
G3	no	Haven't been to a short film festival. Have an interest in short films. Want to join one.	4
G4	no	Don't have interest in short films. Want to attend any events.	5
G5	no	Don't have interest in short films. Don't want to attend any events.	1

---

Observation of Participants’ Behavior at SSF. We had the monitors select two 90-minute programs. We recorded their viewing behavior in four ways: (1) by using a DVD camera recorder located at the back of the theater to capture their film-viewing behavior, (2) by recording their vocalizations with a pin microphone, (3) by installing a small ear-mounted CCD camera to record the scene they were viewing, and (4) by using an electrocardiograph and an accelerometer to capture their physiological responses to the events in the film.

Retrospective Interview. After SSF, we conducted two interview sessions with each study monitor. The first interview session was held on November 7 and 8, 2009; the second one was held on November 14 and 15, 2009. During the interviews, we used the record of the ear-mounted camera and the microphone to show the monitors how they behaved from the first person point of view, and of the image from behind to show the entire view of the scene in order to help them recall what they thought and did at the time their behavior was recorded. These two images with audio were shown at the same time, as illustrated in Figure 1, where the image at the bottom-left is from the ear-mounted camera and the image at the top-right is from the entire-view camera. In this figure, these two images look the same, indicating that this particular study monitor was looking at the screen where the film was projected.

Figure 1. Images of the behavior record used in the retrospective interview.

The first interview was conducted to describe the study monitor’s behavior during SSF and identify the reasons and mental state behind the observed behavior. We focused on three issues: (1) satisfaction of participation in SSF and impressions of the short films that each study monitor watched, (2) each study monitor’s activities in preparation for participating in SSF, and (3) changes in the study monitor’s attitude toward SSF after participating.

The second interview was conducted to clarify the relationship between monitors’ behavior and their preferences. We focused on their attitudes towards short films and film festivals, their preference of films, and how they related the films with their lives (e.g., hobbies, work, school, friends, and communities).

> Results: Autonomic Nervous System Reaction

We measured the electrocardiogram (ECG) of the study monitors while they were watching short films. R-waves were detected from each ECG waveform. Time intervals between two successive R-waves (RR-intervals) were then calculated and recorded using a portable device (ActiveTracer, AC-301A, GMS, Japan). Instantaneous heart rate (HR) [bpm] was obtained by dividing 60 by the RR-interval. The human psychological condition affects autonomic nervous activity (Kreibig, 2010). Therefore, HR increases with sympathetic nerve activation and decreases with parasympathetic nerve activation. The autonomic nervous activity is mainly affected by an individual’s psychological state while watching films at a movie theater, since body movement is minimal. Moreover, since the strength of light and sound of environmental factors other than the films are stable and small in a movie theater, the main factor affecting the autonomic nervous activity is the content of the films.

Monitors Who Exhibited Similar Autonomic Nervous System Reactions. Some monitors exhibited similar autonomic nervous system reactions to the same program. In the following discussion, Monitor 6 (M6) is the key monitor.

M6 watched two short film programs, “Filmmakers B” (F-B) and “National & Local B” (N-B). Three of the fifteen monitors watched program F-B, and six watched program N-B.

Following Yoshino, Matsumoto, Someya, and Kitajima (2011), we calculated the 30-second averaged HR sequence from instantaneous HR data in order to reduce the effect of artifact data. Figure 2 depicts the waveforms of the 30-second averaged HR of M6 (top) and M5 (bottom) watching program F-B. We compared the averaged HR response patterns of the monitors. Figure 3 presents the scatter plots of the 30-second averaged HR sequence of M6 and that of M5. The correlation coefficient between the two was 0.33 (p < 0.0001). Table 2 presents the correlation matrix of the 30-second averaged HR sequence of the three monitors. M6 had the highest correlation with M5 (r = 0.33, p < 0.0001). This result indicates a significant correlation between autonomic nervous activity response patterns to the short film program F-B in M6 and M5.

Figure 2. Thirty-second averaged HR waveform of M6 (top) and M5 (bottom) while watching program F-B with twelve short films.

Table 2. Correlation matrix of the 30-second averaged HR sequence of monitors for program F-B. *: p < 0.0001.

	M6	M4	M5
M6	-	0.08	0.33*
M4	-	-	0.12
M5	-	-	-

This result was replicated for program N-B, as presented in Table 3 (r = 0.32, p < 0.0001). These results suggest M6’s autonomic nervous activity response pattern to short films was most similar to that of M5.

Table 3. Correlation matrix of the 30-second averaged HR sequence of monitors for program N-B. *: p < 0.0001.

	M6	M4	M5	M10	M11	M12
M6	-	0.22	0.32*	0.01	0.10	0.01
M4	-	-	0.22	0.14	0.27	0.07
M5	-	-	-	0.48*	0.37*	0.27
M10	-	-	-	-	0.38*	0.19
M11	-	-	-	-	-	0.26
M12	-	-	-	-	-	-

> Results: Memory of Events

Table 4 briefly summarizes how two study monitors, M5 and M6, remembered the short films they watched in program F-B. They exhibited similar reactions while watching the programs (Figure 3), but they exhibited very different memory in their responses during the retrospective interviews. M6 barely remembered the contents (Table 4). He commented, “I remember nothing” about five of twelve short films. He seemed to remember four films: “Compulsion,” “Butler,” “Checkoo,” and “GAKI Biwa-houshi.” As for the rest, he only vaguely remembered that he had watched the films.

Figure 3. Scatter plots between the 30-second averaged HR sequence of M6 and that of M5 while watching program F-B. The correlation coefficient was r = 0.33 (p < 0.0001).

Table 4. Results of interviews with M6 and M5 about their experiences of program F-B.

Film Title	M6	M5
Director: Michael Wolf
Kontamination: On his way home, an office worker witnesses a horrific event in a parking garage that he will never forget.	I remember nothing.	I remember the contents of the film. It was a grotesque film and made a bad impression. When I viewed the end-roll, I found that many people participated in the creation of the film, even if it was so short. I didn’t have any special impression of the end-roll except for the number of creators. I like suspense. But I don’t understand a film in which people are killed without any meaning.
Externum: It is already very late, when Kathrin comes home. She is tired, but something is wrong. As she tries to open a pillbox, she cuts herself with the kitchen knife. Suddenly, she hears eerie noises coming from the first floor. Something has happened.	I remember nothing.	It was scary because it had many scenes with pain. I didn’t have a good impression of it. I couldn’t watch the scenes with bleeding. I didn’t have any special impression of the end-roll. I just felt it was over. When I watched this film, I was considering that it could have been another scary film. I couldn’t understand the story and had a bad impression of it. From the beginning, the story was jumping and I thought, “Ah! It’s another bloody film!”
The Alchemist’s Book: As a little boy Severin believed in an evil book. Tonight, he will believe again.	I remember nothing.	I woke up here. I had been unconscious until this scene. I remember all the scenes, but they were not impressive. Is it 6 minutes and 48 seconds? I am surprised at the length of the film. I thought it was a longer film. I couldn’t understand the contents well. I don’t think I enjoyed the film because I fell asleep.
Film Title	M6	M5
Director: Erik Rosenlund
Compulsion: Compulsive behavior makes everyday situations seem potentially life-threatening. One day, a man is forced to confront his fears in order to survive.	This film was about mental things. It focused on daily activities. It was interesting but not impressive.	I couldn’t see the words, but when I saw the title afterwards I thought “Oh! I see!”
The Dark Side of the Morning Redux: She should have stayed in bed.	I remember only some scenes. I couldn’t understand the contents. I remember it was uncanny. I don’t remember the story at all, but I like this style.	Was this film about death in a kind of trick play? (viewing the film again) Yes, I remember it. I thought it interesting. It was not outside my interest. This is a scary film, too. But I enjoyed it because the drawing was pretty.
Looking Glass: On a dark stormy night. A little girl is home alone watching television, but is she really the only person there?	I remember nothing.	This is a scary film, too. But I enjoyed it because the drawing was pretty.
Butler: In this light-hearted comedy, a couple find the spark has gone from their marriage. That’s when the butler comes in handy!	I don’t remember this scene. But I remember this film. It was a satirical film with black humor. I pondered over the issue. I found it interesting that the male was in bed and the female was eating in the dining room, and each had his or her own character.	No record of interview
Smalltalk: How do everyday conversations sound to others?	I remember nothing.	No record of interview
Checkoo: An underperforming office worker does everything he can to fit in.	When viewing the scene on the video monitor, I remember the film. It was about the productivity of labor in the past, and there were some scenes that satirized it. I remember the entire film but can’t remember particular scenes. But I enjoyed the film.	I remember this film. The theme was haste. It was interesting because it was from abroad. Foreign animation has a different feeling than domestic animation has. Simply because of this, I felt very interested. At the end, the use of red and yellow patterns was annoying.
Film Title	M6	M5
Director: Reiko Yokosuka
GAKI Biwa-houshi: When Gaki starts to play “biwa”on the veranda, surroundings transform.The gaki is a ”biwa” player of the ogre, who plays the instrument well.When he starts to play it at the veranda, surroundings transform into the profound and remote world.	I remember the title. I remember watching it. The music and picture were unique and speedy. My impression was that it was difficult to understand. The scene was impressive when the biwa player was playing and a kind of surge appeared in the background. I’ve never seen this kind of film. It was mysterious and impressive. I was surprised at the representation. It was interesting. It was new to me, but I didn’t like it.	As I knew this director, I expected this film. It didn’t have a story, but it was amazing that the expression in animation with monochromatic ink painting was completely synchronized with music. I was satisfied with the film and enjoyed it very much.
A piper: The sound of the pipe is suddenly blocked when the piper happily blows it.She blows strongly. And mysterious living things appear from among the pipe.	I have a vague memory of this film. I just remember some scenes.	It was spectacular because one scene changed to another very rapidly. The camera work was astonishing.
Movement: In this light-hearted comedy, a couple find the spark has gone from their marriage. That’s when the butler comes in handy!	I remember nothing.	It was good to see the collaboration with another musician.

> Summary

The findings are summarized as follows. The result of experiencing external events is directly expressed by autonomic responses. Therefore, M6 and M5 looked to have the same experience from watching the short films. At the same time, the experience can be accompanied by another process, memorization, which forms episodic memory of the external events that are consciously recallable. At this point, they seemed to have processed the stimuli very differently: M6 did not process the films in such a way that memory of the films was consciously recallable, whereas M5 did.

Episodic Memory Formation and Two Minds

In this section, we introduce an architecture model, Model Human Processor with Real Time Constraints (MHP/RT), and provide an explanation for the phenomenon we just described.

> Outline of MHP/RT

Toyota and Kitajima (2010) and Kitajima and Toyota (2012) proposed MHP/RT as a simulation model of human behavior selection. It stems from the successful simulation model of human information processing, Model Human Processor (Card, Moran, & Newell, 1983), and extends it by incorporating three theories. The Maximum Satisfaction Architecture (MSA) deals with coordination of behavioral goals, the Structured Meme Theory (SMT) involves utilization of long-term memory that works as an autonomous system, and Brain Information Hydrodynamics (BIH) involves a mechanism for synchronizing the individual with the environment.

MHP/RT includes a mechanism for synchronizing autonomous systems, depicted as round rectangles in Figure 4, working in the “Synchronous Band.” MHP/RT was created by combining two seminal works in different fields that deal with human behavior. The first work is the Model Human Processor in the field of applied psychology (Card et al., 1983). MHP successfully simulates human users’ operating information devices in order to accomplish computerized tasks. The other work focuses on Two Minds that operate in human economical decisions (Evans & Frankish, 2009). This is the basis of the field of behavior economics founded by D. Kahneman, a famous Nobel Prize winner. These two works are combined by applying our conceptual framework of Organic Self-Consistent Field Theory (see Kitajima (2011) for more information).

Figure 4. Two paths.

MHP/RT works as follows: (1) inputs information from the environment and the individual, (2) builds a cognitive frame in working memory, (3) resonates it with autonomous long-term memory, (4) maps the resonance on consciousness to form reduced representation of the input information, and (5) predicts future cognitive frames to coordinate input and working memory.

As depicted in Figure 4, human beings operate in two bands: asynchronous band and synchronous band. The Bodily Coordination Monitoring System and the Memory Processing System operate in the asynchronous band. The Perceptual Information Processing System, Conscious Information Processing System, Autonomous Automatic Behavior Control Processing System, and Behavioral Action Processing System operate in the synchronous band. These systems work autonomously. System 1 of the Two Minds corresponds to the Autonomous Automatic Behavior Control Processing System, and System 2 corresponds to the Conscious Information Processing System.

> MHP/RT and Event Memory

In this section, we use MHP/RT to explain the processes that M5 and M6 carried out while they were watching the short films.

As indicated in Figure 4, two different paths lead to behavioral action and are related to our analysis.

Path 1: The first path goes through Perceptual Information Processing and Autonomous Automatic Information Processing, and directly leads to Behavioral Action Processing. System 1 of the Two Minds determines the behavioral outcome. However, System 2 is not idle. Rather, System 2 may be used to process things that are not related to the film-watching activity (e.g., daydreaming about things that might have been triggered by the film contents but were not strongly related to them).

Path 2: The second path involves the Conscious Information Processing System, whose input is from the Perceptual Information Processing System and whose output will be integrated into the result of the Autonomous Automatic Information Processing System.

As denoted by the dotted oval in Figure 4, the result of the Behavioral Action Processing System is transferred to memory, whose contents reflect how the information has been processed to generate action. Path 1 does not incorporate a conscious process related to the film. Therefore, the memory originating from Path 1 is not associated with the film-watching activity. However, the memory originating from Path 2 includes a memory construct that combines autonomic reactions to the film-watching activity and then-active entities that were consciously activated.

In summary, while watching the short films, M5 formed episodic memory whose contents represents the result of the synchronized integration of Systems 1 and 2; therefore, in the retrospective interview, she could recall what System 2 did when she saw the behavior records, which were tied with memory related to System 1. This could happen because she watched the short films with some concrete purpose, and she felt strong impressions about the contents of the short films. M6 did the opposite of what M5 did, and did not form recallable episodic memory.

Conclusion

We hypothesized that the same experience would not result in formation of the same episodic memory because the experiencing of an event in real time and the formation of memory are different processes performed by Two Minds. Possible differences in the working of Two Minds and in the contents stored in long-term memory affect the formation of episodic memory of the event. We had our monitors watch short films and observed their behavior during the common experience in order to find evidence to support our hypothesis. We demonstrated that the film-watching behaviors of two monitors are consistent with the hypothesis. In addition, we provided an explanation of this phenomenon based on MHP/RT, which operationalizes Two Minds and is capable of simulating such human beings’ behavior as watching films. Further studies will be needed to clarify the workings of Two Minds; MHP/RT will be useful for generating hypotheses and insights into how to design studies to test the hypotheses.

We designed an innovative study method that uses a physical environment consisting of a large projection screen, a set of loud speakers, and a dark closed space, to force the monitors’ perceptual sensors to limit their input to the external stimuli and to concentrate on them. This setup is very different from that in a conventional psychology laboratory. We suggest this design for future research on Two Minds.

References

• Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
• Evans, J. S. B. T., & Frankish, K. (Eds.). (2009). In Two Minds: Dual Processes and Beyond. Oxford: Oxford University Press.
• Kitajima, M. (2011). Organic Self-Consistent Field Theory. http://oberon.nagaokaut.ac.jp/ktjm/organic-self-consistent-field-theory/index.html.
• Kitajima, M., Nakajima, M., & Toyota, M. (2010). Cognitive Chrono-Ethnography: A method for studying behavioral selections in daily activities. In Proceedings of the Human Factors and Ergonomics Society 54th Annual Meeting (pp. 1732-1736). Santa Monica, CA: Human Factors and Ergonomics Society.
• Kitajima, M., Tahira, H., & Takahashi, S. (2010). A Cognitive Chrono-Ethnography study of visitors to a hot-spring resort, Kinosaki-onsen. In M. Kozak (Ed.), Proceedings of the 5th World Conference for Graduate Research in Tourism, Hospitality and Leisure (pp. 436-445). Ankara, Turkey: Detay Anatolia Akademik Yayincilik Ltd.
• Kitajima, M., Tahira, H., Takahashi, S., & Midorikawa, T. (in press). Understanding tourist’s in situ behavior: a Cognitive Chrono-Ethnography study of visitors to a hot spring resort. Journal of Quality Assurance in Hospitality and Tourism.
• Kitajima, M., & Toyota, M. (2012). Simulating Navigation Behavior Based on the Architecture Model MHP/RT. Behaviour and Information Technology, 31(1), 41-58.
• Kreibig, S. D. (2010). Autonomic nervous system activity in emotion: A review. Biological Psychology, 84(3), 394-421.
• Marcus, G. (2008). Kluge: The haphazard construction of the human mind. Boston, MA: Houghton Mifflin Company. Toyota, M., & Kitajima, M. (2010). MHP/RT: Model Human Processor with Real Time Constraints. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (p. 529). Austin, TX: Cognitive Science Society.
• Yoshino, K., Matsumoto, S., Someya, E., & Kitajima, M. (2011). Happiness and heart rate response: a case of fan services at japanese professional baseball games. Natural Science, 3, 255-258.

Top