Syllabus for Economics 308
Agent-Based Computational Economics (ACE):
Growing Economies from the Bottom Up
- Last Updated: 3 June 2024
- Last Course Offering: Spring 2009
- Maintained By:
-
Leigh Tesfatsion
- Professor Emerita of Economics
- Courtesy Professor of ECpE
- Iowa State University
- Ames, Iowa 50011-1070
- https://www2.econ.iastate.edu/tesfatsi/
tesfatsi AT iastate.edu
Course Overview
-
Modern economies are complex systems that can sometimes go
awry --- witness the current financial crisis! How to get a
handle on this complexity?
-
One approach is to model an economy computationally as a
"virtual world" populated by interacting "agents." These
agents can include people, social groupings, institutions,
and/or biological and physical entities.
-
The developer of the virtual world specifies the initial
states of the agents comprising the economy. One objective
might be to study current empirical conditions. Another objective
might be to study hypothetical conditions of interest in order
to see what happens. Once the initial agent states are set, the virtual world
runs forward in time driven by agent interactions, much like
a bacteria culture grows in a laboratory petri dish.
-
Econ 308 introduces students to this exciting new virtual-world
methodology for the study of economic systems. Tentatively
scheduled course topics include:
- decentralized market economies as
complex adaptive systems;
- development and use of computational
laboratories;
- learning and the embodied mind;
- evolution of norms;
- formation of economic trade networks;
- exploration of specific types of market processes (e.g. financial
markets, agricultural markets, energy markets, labor markets, and
automated Internet auctions);
- empirical validation issues.
- As indicated at the following site, agent-based modeling is now supporting scientific research
and technology for a wide variety of commercial applications:
50 Facts About Agent-Based Modeling
(Slides,pdf,6MB)
Topics, Readings, and Exercise Assignments
PLEASE NOTE:
Required materials are marked below with two asterisks (**). Highly recommended
materials are listed with a single asterisk (*) and some recommended materials are listed
with no asterisk. Some modifications to the required and/or recommended materials
might be made as the course proceeds.
Any such modifications will be announced in class and will be marked on the
on-line syllabus with a "new" or "updated" icon for at least one week after
the modification is made.
-
- Introduction
- What are Complex Adaptive Systems (CAS)?
- What is ACE?
- Hands-On Introduction to Agent-Based Computational Modeling
- The Complexity of Decentralized Market
Economies
- Basic Market Concepts
- Market Games
- Learning and the Embodied Mind
- Illustrative Examples of Situated Learning
- Learning Representations
- Application: U.S. Electric Power Market Restructuring
- Application: Financial Markets
- Interaction on Fixed Networks
- Formation of Interaction Networks
- Empirical Validation of ACE Models
- Appendix:
General Course Project Information
I. Introduction
I.A What are Complex Adaptive Systems (CAS)?
- Key In-Class Discussion Topics:
- What is a complex system? a complex adaptive system?
- Illustrative examples (Cellular automata, Bak's Sand Pile Model, Schelling's
Segregation Model,...)
- Experimental design: Basic concepts and terminology
- Important: Please note that, as indicated at the top of the exercise assignment, late assignments will not be accepted -- no exceptions. An assignment is late if it is turned in after discussion of the exercise answers has commenced on the due date. If you cannot attend class on the due day, either give your exercise to a classmate for turning in or put your exercise under the instructor's office door (Heady 375) no later than 10:45am on the due date. Do not leave exercises in mailboxes or send them via email except by pre-arrangement, since they might not be received in time.
- Required Readings:
- **
Tamás Vicsek,
"Complexity: The Bigger Picture"
(Paper,pdf,71KB),
Vol. 418, 11 July 2002, p. 131.
- Abstract:
In this short essay, Vicsek describes how computer simulation fits
into the scientific enterprise. The goal is to "capture the principal laws
behind the exciting variety of new phenomena that become apparent when the
many units of a complex system interact.
- ** Andy Clark, "Preface: Deep Thought Meets Fluent Action"
(pp. xi-xiii) and "Introduction: A Car with a Cockroach Brain"
(pp. 1-8), in Being There: Putting Brain, Body, and World Together
Again, MIT Press, 1998 (paperback). HAND-OUT
- Note: If at all possible, some day make time to savor this entire
delightful book!
- ** Leigh Tesfatsion, "Notes on Batten Chapter 1 - stress on the Glossary of Terms"
(Notes,html).
- ** Leigh Tesfatsion, "Possible definitions for `Complex System' and
`Complex Adaptive System'"
(Slides,pdf,19KB).
-
** Leigh Tesfatsion, "Introduction to Cellular Automata"
(Slides,pdf,1MB).
-
** Leigh Tesfatsion,
"Implementing Per Bak's Sand Pile Model
as a Two-Dimensional Cellular Automaton"
(Notes,pdf,50KB).
-
**
Leigh Tesfatsion,
"Experimental Design: Basic Concepts and Terminology"
(Notes,pdf,45KB).
- Recommended Materials:
-
* Game of Life, Sand Pile Model, and Schelling Segregation Model: Demonstration Software
(html)
- Interactive computational frameworks for running hands-on
experiments with John Conway's Game of Life, Per Bak's Sand Pile Model, and Thomas Schelling's Segregation
Model (as well as cellular automata more generally) can be found at this site.
- * David F. Batten, "Preface" plus Chapter 1:"Chance and Necessity"
(preprint(no figs),pdf,247KB)
plus Chapter 8: "Artificial Economics"
(pdf preprint-no figs,173KB)
in Discovering Artificial Economics: How Agents Learn and Economies Evolve, Perseus Books, Westview Press, 2000.
- IMPORTANT NOTE: The Batten book is unfortunately out of print. However, a pdf file for the entire Batten book (including figures) can be accessed
here (pdf,17MB).
-
*
Peter Albin, Preface (pp. xiii-xxxi)
(Paper,pdf,146KB)
and Duncan K. Foley, Chapter 1: "Introduction (pp. 3-22)
(Paper,pdf,369KB),
in Peter S. Albin and Duncan K. Foley (Eds.), Barriers and Bounds to Rationality: Essays on Economic Complexity and Dynamics in Interactive Systems, Princeton Studies in Complexity, Princeton University Press, NJ, 1998, posted with permission of Princeton University Press.
- Abstract: This book preface and pages 3-22 of this wide-ranging introductory chapter by two seminal contributors to economic complexity theory discuss possible automata-theoretic resolutions to economic complexity puzzles and an overview of dynamical systems in both the social and physical sciences.
I.B What is Agent-based Computational Economics (ACE)?
- Key In-Class Discussion Topics:
- What is ACE all about?
- Illustrative example
- Required Readings:
- Recommended Materials:
- * Robert Axelrod and Leigh Tesfatsion, "A Guide for Newcomers to Agent-Based Modeling
in the Social Sciences"
(html).
- * Rob Axtell, "Agent-Based Computing in Economics"
(Slides,pdf,256KB),
a more advanced discussion of ACE presented at the VII Trento Summer School on ACE, July 2006.
-
*
Duncan K. Foley, Chapter 1: "Introduction (pp. 23-72)
(Paper,pdf,369KB),
in Peter S. Albin and Duncan K. Foley (Eds.), Barriers and Bounds to Rationality: Essays on Economic Complexity and Dynamics in Interactive Systems, Princeton Studies in Complexity, Princeton University Press, NJ, 1998, posted with permission of Princeton University Press.
- Abstract: Pages 23-72 of this wide-ranging introductory chapter by a seminal contributor to economic complexity theory covers the following topics: Economic complexity puzzles; economic models of fully rational behavior; definitions and measures of complexity; complexity in cellular automata; modeling of complex social and economic interactions; complexity, rationality, and social interaction; and towards a robust theory of action and society.
-
Other introductory source materials on CAS/ACE
I.C Hands-On Introduction to Agent-Based Computational Modeling
- Key In-Class Discussion Topics:
- What is "object-oriented programming (OOP)"?
- What's the difference between an "object" and an "agent"?
- Availability of software modeling tools for Agent-Based Modeling (ABM)
- Which ABM software modeling tools are best for you?
- Template ABM models for getting started
- Should you use an Integrated Development Environment (IDE)?
- Exercise:
** Exercise 3 (Individual - Pass/Fail): "Hands-On Introduction to Agent-Based Modeling"
(pdf,27KB),
Due: Tuesday, February 17, 11:00am.
- Required Readings:
-
** Leigh Tesfatsion,
"Introduction to Agent-Oriented Programming"
(Slides,pdf,110KB).
-
Note:
This tutorial briefly discusses basic object-oriented
programming (OOP) concepts, what is agent-oriented programming (AOP), and how
AOP compares and contrasts with OOP. It also briefly discusses
how AOP applications can be implemented via computational laboratories, using the Trade Network Game (TNG) Laboratory
(html)
for concrete illustration.
- ** Nicholas R. Jennings,
"On Agent-Based Software Engineering"
(pdf,257KB),
Artificial Intelligence 117 (2000), 277-296, copyright © 2002
Elsevier Science B.V. All rights reserved.
- Recommended Materials:
-
* The Trade Network Game Lab (C++/VB, open source), demonstration software for market games, trade network formation, & GA learning, by McFadzean, Stewart, and Tesfatsion
(html)
- * Matt Weisfeld, "Introduction to Object-Oriented
Concepts" Chapter 1 in The Object-Oriented Thought Process,
SAMS Books, Macmillan, Second Edition, 2003.
- * Rob Axtell, "Platforms for Agent-Based Computational Economics"
(Slides,pdf,35KB),
presented at the VII Trento Summer School, July 2006.
-
* Alan G. Isaac, "The ABM Template Models: A Reformulation with Reference Implementations"
(html),
Journal of Artificial Societies and Social Simulation 14 (2) 5, March 2011.
- Abstract: The author refines materials originally developed by Steven F. Railsback, Steven L. Lytinen, and Stephen K. Jackson to guide students, step by step, through a hands-on construction of an ACE model using sixteen model templates incorporating increasing capabilities. The author also addresses design, flexibility, and ease of use issues relevant to the choice of an agent-based modeling platform.
- * William Rand, Agent-Based Modeling Platforms: A Practical Introduction
(pdf,4.9MB),
presented at ISU, January 30, 2007.
-
ABM General Software and Toolkits
-
ABM Computational Laboratories and Demonstration Software
II.
Complexity of Decentralized Market Economies
II.A Basic Market Concepts
- Key In-Class Discussion Topics:
- Modeling decentralized market economies
- Key types of market players
- Key types of market structures
- Construction of demand and supply functions
- Competitive vs. Strategic Pricing
- Exercise:
** Exercise 4 (Individual, 20 Points): "Competitive Versus Strategic Pricing"
(pdf,123KB),
Due: Tuesday, February 24, 11:00am.
- Required Readings:
-
** Leigh Tesfatsion, "ACE Market Modeling: A Short Introduction"
(pdf,325KB).
-
** Leigh Tesfatsion,
"Market Organization with Price-Setting Agents"
(Notes,html,8KB).
-
** Leigh Tesfatsion, "Market Basics for Price-Setting Agents"
(Notes,pdf,422KB).
- Note: These "Market Basics" notes should be read together with the following notes on demand and supply schedule construction.
-
** Leigh Tesfatsion, "Illustrations of Demand & Supply Schedule Construction"
(Slides,pdf,168KB).
- Recommended Materials:
-
* Leigh Tesfatsion,
"Notes on Price Discovery with Price-Setting Agents"
(pdf,103KB).
-
*
ACE-Related Research on Multi-Market Modeling
II.B Market Games
- Key In-Class Discussion Topics:
- Basic game theory concepts
- Market games among multiple learning traders
- Can market structure substitute for trader rationality?
- Exercise:
** Exercise 5 (Team/Individual, 14 Points): "Zero-Intelligence Market Trading Exercise"
in three versions corresponding to three different agent-based toolkits, as follows:
-
MASON version (25KB)
-
RepastJ version (38KB)
-
NetLogo version (25KB)
- Required Readings:
-
** Leigh Tesfatsion,
"Game Theory: Basic Concepts and Terminology"
(NotesAndSlides,pdf,34KB).
-
** Leigh Tesfatsion,
"ACE Market Game Examples"
(Slides,pdf,289KB).
-
** Dhananjay K. Gode and Shyam Sunder, "Allocative Efficiency of Markets with Zero-Intelligence Traders: Markets as a Partial Substitute for
Individual Rationality"
(pdf,1.4MB),
Journal of Political Economy, Vol. 101, No.
1, 1993, 119-137.
-
**
John Duffy, Notes on Gode-Sunder Zero-Intelligence Traders
(pdf,719KB).
- Recommended Materials:
-
* Leigh Tesfatsion,
"Price Discovery with Price-Setting Agents (Market Games)"
(pdf,103KB).
III. Learning and the Embodied Mind
III.A Illustrative Examples of Situated Learning
- Key In-Class Discussion Topics:
- Does logic prevent cooperation?
- How do people learn in settings with multiple decision makers?
- How do people make trade-offs between own payoffs and a concern for fair play?
- Required Readings:
-
** Leigh Tesfatsion,
"Notes on Axelrod's IPD Tournaments"
(Notes,pdf,473KB).
- ** Douglas R. Hofstadter, "Computer Tournaments of the Prisoner's Dilemma Suggest How Cooperation Evolves"
(pdf,856KB),
Scientific American, May 1983, 18-26.
- Abstract: Hofstadter explains Robert Axelrod's computer tournaments, which explored the evolution of cooperation in the Iterated Prisoner's Dilemma. For the original work, see above.
- Recommended Materials:
-
* Axelrod Tournament Demonstration Software (C#, open source), by Chris Cook
(html)
-
* Robert Axelrod (1984), The Evolution of Cooperation (Chapters 1,2,9)
(pdf,3.6MB),
Basic Books Inc., New York, NY.
III.B Learning Representations
- Key In-Class Discussion Topics:
- What can be inferred from Rodney Brooks' observation that
"elephants don't play chess"?
- Are people's "minds" best viewed as disembodied logical
reasoning devices or as controllers for embodied activity?
- Illustrative learning representations (e.g., reinforcement learning,
Q-learning, genetic algorithms (GAs), GA-classifier systems, artificial
neural networks, ...)
- How should learning be represented for economic agents?
- If you had to construct a computational firm or consumer capable of
surviving over time in a virtual economy, let alone prospering, how would you do it?
- Should the cognitive processes of computational agents
necessarily mimic the cognitive processes of real
people?
- Take-Home Exercises:
- Required Readings:
- Recommended Materials:
-
* The Trade Network Game (TNG) Laboratory (C++/VB, open source) with Genetic Algorithm (GA) trader
learning, by McFadzean, Stewart, and Tesfatsion
(html)
- * Andy Clark, Chapter 9: "Minds and Markets" (pp. 179-192).
- * Leigh Tesfatsion,
"Notes on Clark Chapter 9"
(html).
- * David F. Batten, Chapter 2:"On the Road to Know-Ware"
(preprint(no figs),pdf,224KB),
in Discovering Artificial Economics: How Agents Learn and Economies Evolve, Perseus Books, Westview Press, 2000. ONLINE
- * Douglass C. North, "Economics and Cognitive Science"
(pdf,18KB),
Working Paper, Washington University at St. Louis, 1996.
- This paper focuses on a key unresolved puzzle (also addressed by
Andy Clark): How do humans evolve "scaffolding" (internal belief systems and
external institutions) to reduce the uncertainty coming from the strategic
interaction of human beings in economic, political, and social market
situations? Douglass North is the 1993 recipient of the Bank of Sweden Prize
in Economic Sciences in Memory of Alfred Nobel.
-
* Mridul Pentapalli,
"A Comparative Study of Roth-Erev and Modified Roth-Erev Reinforcement Learning Algorithms
for Uniform-Price Double Auctions"
(Slides,6.5MB),
M.S. Computer Science Thesis, ISU, March 2008)
-
* Bill Smart, Reinforcement Learning: A User's Guide
(Slides,pdf,430KB),
Washington University, St. Louis.
-
Other source materials related to learning
IV. Application: U.S. Electric Power Market Restructuring
- Key In-Class Discussion Topics:
- How are U.S. wholesale power markets currently being restructured?
- How might ACE frameworks be used to test the efficiency, reliability, and fairness of the designs
being proposed for restructured wholesale power markets?
- Required Readings:
-
** Leigh Tesfatsion,
"Stress-Testing Institutional Arrangements via Agent-Based Modeling: Illustrative Results for U.S. Wholesale Power Markets"
(Slides,pdf,2MB).
- Recommended Materials:
-
* AMES Market Package (Java, open source), a test-bed for the agent-based modeling of electricity systems, by Li, Sun, and Tesfatsion
(html)
-
Other source materials related to ACE Electricity Research
-
General resources on electricity restructuring
V. Application: Financial Markets
- Key In-Class Discussion Topics:
-
- What makes financial assets/markets special?
- What is the "efficient markets hypothesis (EMH)"?
- Do sophisticated (informed) traders necessarily drive "noise traders"
from financial markets?
- How should the market mechanisms governing financial asset trading be
modelled?
- How should the learning processes of financial traders be represented?
- How should the "fitness" of financial traders be measured? What drives
the co-evolution of these fitnesses over time?
- What are the basic "empirical stylized facts" for financial markets? How
well are these stylized facts captured by standard financial models?
By agent-based financial models?
-
- Required Readings:
-
-
** Leigh Tesfatsion, "Stock Market Basics"
(Slides,pdf,277KB).
-
** Silvano Cincotti, "Some Stock Market Stylized Facts"
(Slides,pdf,615KB)
-
** Leigh Tesfatsion, "The Santa Fe Artificial Stock Market: Overview"
(Slides,pdf,99KB)
-
** Leigh Tesfatsion, "Rational Expectations, the Efficient Market
Hypothesis, and the Santa Fe Artificial Stock Market Model"
(Slides,pdf,856KB).
-
-
Recommended Materials:
-
-
Santa Fe Stock Market Demonstration Software
- * Rob Axtell, "ACE Financial Market Modeling"
(Slides,pdf,82KB),
presented at the VII Trento Summer School on ACE, July 2006.
- * David F. Batten, Chaper 7: "Coevolving Markets"
(Preprint(no figs),pdf,247KB),
in Discovering Artificial Economics: How Agents Learn and Economies Evolve, Perseus Books, Westview Press, 2000.
- * Blake LeBaron, "Building the Santa Fe Artificial Stock Market"
(Paper,pdf,123KB),
Working Paper, Brandeis University, June 2002.
- Abstract: This brief summary provides an insider's look at
the construction of the Santa Fe Artificial Stock Market (ASM) model. The
perspective considers the many design questions that went into building the
model from the perspective of a decade of experience with agent-based
financial markets. The model is assessed based on its overall strengths and
weaknesses.
- * Leigh Tesfatsion,
"Detailed Notes on the Santa Fe Artificial Stock Market Model"
(html).
-
Other source materials related to ACE financial modeling
VI. Interaction on Fixed Networks
- Key In-Class Discussion Topics:
- What might be inferred from the observation by Craig Reynolds that
"a flock is not a big bird"?
- Distinguishing between "simple" and "complex" economic systems
- Under what circumstances can robust point predictions of economic
outcomes be obtained from a knowledge of initial economic structure,
ignoring network effects? And when might network
effects be important for the prediction of economic outcomes?
- How can graph theory be used to quantitatively represent and analyze
economic interaction networks?
- What type of systematic phase transition do random graphs undergo as
their connectivity increases?
- Do socioeconomic networks exhibit any kind of systematic phase transition
as their connectivity increases?
- Why all the recent excitement about "small-world networks"
(locally dense networks with global reach)?
Required Readings:
-
** Leigh Tesfatsion, "Introductory Notes on the Structural and Dynamical Analysis of Networks"
(Slides,pdf,2.3MB).
- NOTE: These presentation slides summarize and graphically illustrate key points from the
"Introduction to Networks" notes linked below.
-
** Leigh Tesfatsion, "Introduction to Networks"
(html).
- Abstract: These notes provide rigorous definitions for basic structural characterizations of networks (e.g., degree, clustering, shortest path length). Also discussed are phase transitions in random graphs, the concept of a "small world network," and the possible application of small-world networks to the study of trade interactions. The Key references are Batten (Chapter 3, 2000) and Wilhite (2001), both linked below.
-
** Leigh Tesfatsion, "Notes on Wilhite (2001)"
(Slides,pdf,236KB).
- NOTE: These presentation slides summarize key points from the article by Wilhite (2001), linked below.
- ** Allen Wilhite (2001), "Bilateral Trade and `Small-World' Networks"
(Paper,pdf,181KB),
Computational Economics, Vol. 18, No. 1, August, pp. 49-64.
- Abstract: Wilhite develops an agent-based
computational model of a bilateral exchange economy in which profit-seeking traders sequentially engage in trade partner search, negotiation, and trading. He uses this model to
explore the consequences of restricting trade to different types of networks,
including a "small-world network" with both local connectivity and global
reach. His key finding is that small-world networks provide most of the
market-efficiency advantages of completely connected networks while retaining
almost all of the transaction cost economies of locally connected networks.
- Recommended Materials:
- * David F. Batten, Chapter 3: "Sheeps, Explorers, and Phase Transitions"
(Preprint(no figs),pdf,203KB),
in Discovering Artificial Economics: How Agents Learn and Economies Evolve, Perseus Books, Westview Press, 2000.
- * Steven H. Strogatz, Exploring Complex Networks
(pdf,589KB),
Nature, Vol. 410, 8 March 2001, pp. 268-276.
-
Other source materials related to ACE network research
VII. Formation of Interaction Networks
- Key In-Class Discussion Topics:
- In what economic situations are interactions determined randomly over time?
- In what economic situations are interactions determined preferentially over time by choice
and refusal of trade partners based on past experiences?
- What difference might it make if economic interactions are randomly versus preferentially determined?
- A labor market study illustrating preferential network formation among workers and employers with learning capabilities
- Representation and visualization of network formation: How should it be done?
- Required Readings:
- ** Leigh Tesfatsion, "Notes on Network Formation"
(Slides,pdf,246KB).
- ** Leigh Tesfatsion, "Illustrative Application: Labor Institutions and
Market Performance"
(Slides,pdf,117KB).
- Recommended Materials:
-
* The Trade Network Game (TNG) Laboratory (C++/VB, open source), includes run-time visualization of trader network formation, by McFadzean, Stewart, and Tesfatsion
(html)
- * Albert-László Barabási, "Network Overview"
(Slides,pdf,70MB),
2006 Keynote Address. (Caution: Large download)
- Professor Albert-László Barabási (Department of
Physics, Notre Dame, Indiana) directs a research group focusing on the
emergence and evolution of networks in various contexts (e.g., metabolic and
genetic networks, actor networks, collaborative networks). This fun slide
presentation provides a vivid visual summary of some of their key findings to date.
- * David F. Batten, Chapter 4:"The Ancient Art of Learning by Circulating"
(pdf preprint - no figs,167KB),
in Discovering Artificial Economics: How Agents Learn and
Economies Evolve, Westview Press, Boulder, Colorado, 2000, plus
Leigh Tesfatsion, "Notes on Batten Chapter 4, Plus Glossary of Terms"
(html).
-
Other source materials related to ACE labor research
-
General resource site on network formation
VIII. Empirical Validation of ACE Models
- Key In-Class Discussion Topics:
- Verification for ACE models: How to verify an ACE model is carrying out operations in the way the modeler intends?
- [G.E.P. Box (1979)]: "All models are wrong, but some are useful." Must the intended purpose of a model be known before meaningful empirical validation can proceed?
- Empirical validation for ACE models: input validation (operational validity), descriptive output validation, and predictive output validation
- What is iterative participatory modeling (companion modeling)?
- Required Readings:
- ** Leigh Tesfatsion, "Notes on the Empirical Validation of ACE Models"
(Slides,pdf,176KB).
- Recommended Materials:
- *
Paul Windrum, Giorgio Fagiolo, and Alessio Moneta,
Empirical Validation of Agent-Based Models: Alternatives and Prospects
(html),
Journal of Artificial Societies and Social Simulation, Vol. 10, no. 2,8, March 31, 2007.
- Abstract: This paper addresses the problem of finding the appropriate method for conducting empirical validation in ACE models. The paper has two primary objectives: (1) to identify key issues facing ACE economists engaged in empirical validation; and (2) to critically appraise the extent to which alternative approaches deal with these issues.
-
Other source materials on the empirical validation of ACE models
Appendix: General Course Project Information
Students are strongly encouraged to begin consideration of possible course project topics as soon as possible.
Please visit the
Course Project Information Site
for detailed information regarding course projects, including a list of
course projects selected by Econ 308 students in previous years.
I am available during office hours, by appointment, and anytime by email to
provide guidance if desired.
Preliminary outlines for student project proposals must be turned in to the instructor during the first week following Spring break and must receive go-ahead instructor approval by the end of March.
Final write-ups for student project reports are due the last day of
class.
Copyright © Leigh Tesfatsion. All Rights Reserved.
