The Digital Ethnographer’s Workbench project is part of a growing movement in the behavioral sciences to reengage a set of questions that were abandoned early in the cognitive revolution. These questions have to do with the nature of human activity as it occurs in natural settings. We are looking where cognitive science has not paid much attention: fine-scale details of moment-by-moment interactions. We approach this with a theoretical framework that is still being developed. Recent advances in recording technologies give us access to the details of human interaction. Our first order of business is to understand what the phenomena are. What needs explanation? What do people actually do? In our very brief presentations to the lab in recent weeks we have encountered some really interesting phenomena. A partial list includes the following items. The comments in italicsindicate discussion topics that I hope to pursue in the lab meetings.
These phenomena are only weakly represented in contemporary cognitive science. It seems incredible to me that cognitive science knows so little about the phenomena of real world cognition. We might ask ourselves why this is so. One sort of explanation goes like this: Theory drives choices about where to look and what to look for. This looking in turn constrains what one sees and what one thinks needs to be explained. And this sense of what exists and what needs to be explained drives the development of theory. This is a cycle and every scientific field spirals around it, accumulating phenomena, explanations, understandings, and ways of achieving understanding. A scientific field is a complex dynamical system that can settle into trajectories that make some kinds of phenomena central and marginalize others. The history of our field has made so called “internal cognitive processes” the central objects of concern. This move marginalizes culture, context, history, and affect. In CITW I make the claim that founding cognitive science on the pursuit of internal symbolic processes was a mistake. The overthrow of the physical symbol system hypothesis in the past decade has led to a widening of the range of phenomena considered legitimate objects of cognitive scientific scrutiny. Yet, the things that we cognitive ethnographers are looking at are still on the periphery of the field. We are working far from the center of the cognitive science. Whether we are far in the lead (as I hope), or far out in left field is something that will be decided by history.
Methods play an important role in this dynamic process. As methods deliver results, they become legitimated. When cognitive science was being born in the 1950s, the midwives of the field faced a difficult practical problem. They felt that the developments in information theory, computer technology, and neuroscience gave them license to depart from the previously dominant behaviorism and to speak of internal psychological processes. But the constraints of the politics of science dictated that this discussion had to be completely different from the introspective methods of Gestalt psychology. There was a good deal of science envy around. It was felt necessary to have the new science of mind be scientific. Without doubt this science envy promoted methodological rigor, but it also had some pernicious effects. Methodological rigor is a powerful motive to study that which is easy to quantify (e.g., reaction times) and to ignore that which is difficult to quantify (e.g., meaning, or action in a real world). This does not mean that rigorous approaches to more difficult problems cannot be developed. It simply means that doing so requires more thought and effort.
The discourse of scientific rigor can have poisonous effects at two levels integration in our community. We can see effects at the level of an entire community in the history of cognitive science and cognitive anthropology. The effects are felt in a different way in local communities, such as our own department and laboratory.
In the field as a whole the impatient pursuit of rigor leads to a focus on that which is easy to measure. I can illustrate this with a story about my own field, cognitive anthropology.
For cognitive anthropology, the theoretical commitment to internal phenomena meant first a focus on mental contents in the form of knowledge, rather than on action or organization of the environment of thought. Cognitive anthropology then abandoned efforts to understand the social world and the material world. The perceived need to make the research methods appear “scientific” strongly biased the choices towards methods that were quantitative and replicable (to counter the crisis caused by different ethnographers bringing home very different stories about the same society). Borrowing tools from the then expanding formal linguistics, the focus of cognitive anthropology contracted from all knowledge to knowledge expressed in language. This was justified by noting (falsely) that most cultural knowledge is transmitted via language. (Imagine the program of research that would be required to evaluate this claim. One would surely have to make a systematic study of real world instances of the transmission of knowledge. No such project was ever undertaken, but the claim was accepted.) The move was made and anthropologists became interested in imagining internal structures and writing formal rules that could transform those imagined structures into (a very limited set of) observed language behaviors.
One of the lessons here is that good science always keeps its eye on the phenomena of interest. As one settles into a solution in which a theory and methods and phenomena can be kept in contact with one another, we have to ask how much compromise we are willing to accept in each area. One hallmark of good science is sustained focus on the phenomena. Good scientists do not let methodological convenience draw them away from the phenomena of interest. But it does happen.
Later, borrowing the techniques of multidimensional scaling from Roger Shepard in psychology, the focus of cognitive anthropology contracted again. By the late 1960s cognitive anthropology had become the study of the semantic structure of noun domains. From this we learned about latent semantic structure in kinship terminologies, folk classifications of the living world, kinds of firewood, body parts, illnesses, psychological states, etc. That is all fine and useful work, but look at what was left out. The study of how such knowledge is represented and used in real world activity was trampled and cast aside in the rush to appear scientific. This trend was repeated in all of the “cognitive sciences” and is precisely why so little is known today about how cognition works in real world settings. Anxiety about how their methods looked to other scientists (and we cannot forget that the other scientists held the purse strings of the research funds) led cognitive anthropologists to abandon the study of the most interesting phenomena within their reach. This is why we must strive to build a new approach to cognition that does not back down in the face of the difficulties encountered in trying to understand cognition in the wild. As Morana Alač and I say in the introduction of a new paper about cognition in scientific practices,
“The cognitive processes that play out in the interactions of human actors with the social and material world are a previously under-appreciated domain of cognitive phenomena awaiting exploration by cognitive anthropologists. Because these processes are embedded in the culturally constructed environment of scientific practice, they are at once fundamentally cognitive and especially amenable to anthropological approaches. The careful examination of these interactions reveals cognition in action. We see this approach as a new kind of cognitive anthropology.”
At the moment, cognitive anthropology is moribund. It has exhaustively mined myriad noun domains using the quantitative methods of MDS. It has since embraced schema theory as a formalism, and has blithely followed psychologists in attributing schemas as unobserved internal mental constructs. Cognitive anthropology was already fatigued when the post-modernists attacked it for being (paradoxically) both irrelevant and a form of violence against other cultures. Few students are being trained as cognitive anthropologists today. How sad that is when an entire world of rich cognitive phenomena lies on the surface of the mind waiting to be understood. And how sad that cognitive science has made such great leaps, but still cannot bring itself to look directly at what it originally set out to understand.
The “is what you do really science?” discourse affects local communities according to a different social dynamic. Jim and I have long recognized that our students operate from a disadvantaged position in the department. Our interests lie in areas that are not at the core of cognitive science. Anthropology is a marginalized discipline, having made few recognized contributions to cognitive science, and HCI bears the stain of being applied rather than pure science. Of course, Jim and I do not believe that what we do is peripheral to the interests of the field. We know that since the real world is not well understood, a huge amount of pure science needs to be done there, and we recognize that building, implementing, and installing a system in a real world setting can be a better test of a theory than any sequence of carefully designed laboratory experiments. We have set out in the lab to construct a new foundation for human machine interaction based on a new foundation for cognitive science as a whole. We invite like-minded students to join us, but unlike more established fields, we usually cannot hand them a project off the shelf with a crank to turn that will spit out publishable results.
In a local community of budding scholars struggling to understand phenomena in an area where neither theory nor method has yet been consolidated and institutionalized, the “is it science?” talk has a chilling effect. It sets up a phony litmus test for legitimacy of the work. My students are asked by their colleagues, “What is your experiment?” The presuppositions behind this question are huge, unexamined, and not in the least understood by the people posing the question. What the posers (both meanings intended) do understand is the social power of that question to marginalize others when invoked in a community that cherishes the trappings of experimental science. This discourse is hostile, and is often intended to be so.
I categorically reject the proposition that our choices are either 1) to make what we do look like other cognitive sciences or 2) abandon the claim to be doing science. A lot of really awful non-science is done by people who are paying most attention to looking like they are doing science. Fortunately, there is a third path. We can do excellent science of a different sort of phenomena. We must face the fact that the phenomena we are investigating have not received much scientific attention and are not well understood. Simply arguing that these phenomena are legitimate objects of scientific scrutiny is still an uphill battle in the cognitive science community. Since the phenomena are different, we should not expect that the methods that serve the investigation of other sorts of phenomena will be most useful. Consider the fact that the theoretical objects of most interest to classical cognitive science, internal mental processes, cannot be observed directly. Since they cannot be observed, their nature must be inferred from things that can be observed. This is one reason why experimental methods are so important to cognitive science. A well-designed experiment can make invisible processes visible3. Notice that we are in a very different situation with the phenomena that I listed at the top of this note. These things can be observed directly. Our first problem is not to infer the existence of an imagined, but unobservable, process. Rather the first problem we face is to find the patterns in complex processes that are directly observable. In this sense, our current core enterprise is more like meteorology, or ecology, or even descriptive linguistics, than it is like cognitive psychology. These are perfectly good kinds of science, and experiments can play a role in testing theories in them, but they are not principally experimental in nature. Formal models tend to play a much more important role than experiments in sciences that seek to understand the patterns or law-like relations in directly observable complex systems. Thus, asking DEW to manifest the trappings of some other science is exactly the wrong strategy.
Of course we should have some idea of what makes something science. Here is an abbreviated account of what I think makes good science.
1. Our purpose is to understand how some aspect of the world works. It is important to be clear about what aspects of the world we care about.
2. Responsibility to the phenomena. The data have the last word. Theory must conform to the world. This is why manipulating the data to fit one’s theory is the greatest crime in science. It is also extremely important to be explicit about the range of phenomena to which we hold our theory responsible.
3. Realize that we (as scientists and as folk) are at risk of being fooled by our own pretty stories. This means that we must have a plan for managing these risks. This is where discussions of method belong. Method must always be in the service of understanding the phenomena. Abstract discussions of the virtues of various methods in the absence of specific epistemological problems are nonsense.
4. Theory and phenomena must be brought into contact via the application of methods of investigation. The opportunity to reject theory must exist. You might be surprised how often sophisticated methods are applied to phenomena in ways that have no bearing on any theoretical question.
5. We must examine the concepts that we use to decide the meanings of our findings.
I believe in these five principles. I try to hold my own work and the work of my students to these standards. But I also try to teach this by doing it rather than by talking about it.
In my plenary address to the cognitive science society last August, I presented five contributions that cognitive ethnography can make to cognitive science. All five of them are relevant to the current discussion.
1. Understanding cognitive ecology: this is an anti-reductionist move. Rather than decomposing the person/culture/environment system into smaller and smaller structural pieces, we attempt to examine functional constellations. This leads to lots of crossing of traditional disciplinary and analytic boundaries. The focus is on interactions and process instead of structure. The phenomena listed at the beginning of this note become visible in this perspective. Culture, context, and history, which were intentionally marginalized in the beginnings of cognitive science, now find natural roles to play in the organization of cognitive processes.
2. Providing a new functional specification for the human mind: Because it has never taken real world cognitive activity seriously as an object of scientific scrutiny, cognitive science has little knowledge of what minds actually do. Here is the source of my skepticism about the standard list of underlying cognitive processes. Seeing how cognition is embedded in the social and material environment shifts our understanding of what a mind must be in order to do what it actually does.
3. Documenting the distribution of cognitive processes across space, time, and society. What is universal in human thinking, what is pervasive, what is common, what is rare? The short history of cognitive ethnography has established important commonalities across activity settings. Opportunism, interaction with material and social environment, complex adaptation processes that span the brain, the body, and the world. There are also important differences that follow from differences in the availability and use of tools (including cognitive technologies such as literacy), different sorts of social organization, attitudes toward thinking, metacognition, etc. How much thinking happens in rich interaction with a culturally elaborated material and social world, and how much is more like the disembodied rationation that has been the default assumption of cognitive science? I will guess it is the former, and you may guess it is the latter, but the point is that until we have good systematic studies of real world cognition, we are all guessing.
4. Inform experimental studies in several ways. Cognitive ethnography can generate interesting hypotheses to be tested using experimental methods. It can provide useful input to the design of stimulus materials. It can help to evaluate the ecological validity of experimental studies. Discussions of ecological validity are empty without detailed knowledge of the cognitive ecology from which the subjects are drawn. It would be nice to avoid the tragic waste of resources on experiments that have nothing to do with actual cognitive processes.
5. Inform the design of artifacts. Over the years, Jim and I have had great success in basing the design of tools and training systems on cognitive ethnographic studies of real world settings.
Each of these is a legitimate contribution to the scientific study of cognition. We should aspire to the creation of a laboratory that has a coherent program to address all of these issues. Of course we are still learning how to do it, but I firmly believe that our laboratory is the best thing going today.
At the level of a scientific field I imagine this project to be doing the following.
1. Expanding the horizons of cognitive science by focusing on the phenomena that should be of most interest to the field: real world cognitive activity.
2. Understanding what those phenomena are and what we can do to improve our understanding of them.
3. Developing theoretical accounts that explain and allow us to predict what happens in those activities. THIS IS THE CHALLENGE OF DISTRIBUTED COGNITION THEORY. Ultimately we hope to provide explanations that connect with the rapidly expanding body of knowledge about the nervous system, perceptual and motor processes, and the body in the world. For reasons too complex to explain here, I am skeptical about connections to the established literatures on so-called “high-level” cognitive processes.
4. Adapting and developing tools and methods as appropriate for the challenges of testing theoretical conjectures about the nature of real world cognition. THIS IS THE DEW MISSION. New technologies give us the opportunity to see new things and to understand them in new ways. That’s incredibly exciting! Think of this in terms of the cycle of scientific development described above. New tools, new observations, new phenomena, new theory, new tools….
5. Articulating the new view of cognitive science in a way that makes it available and useful, first to like-minded ethnographers of real world cognition, and then to the larger cognitive science community. Over the years I have learned to be patient. Scientific communities are conservative by nature. That’s a good thing, but it means that changes happen slowly.
Finally a few reflections on the training of students in this area.
In our lab and in our lab meetings I imagine myself to be doing the following:
1. Recognizing that there is a rapid turnover of personnel in the lab. Each year new people join us while more experienced students receive their Ph.D. degrees and leave us. This means that most participants in the lab meetings have read only a small fraction of my work or Jim’s work, or the literature in cognitive science, for that matter. I often feel like I am perpetually starting over from scratch. But that’s a virtue as well, because each time we learn to talk to each other, we learn new ways to talk about what we are learning.
2. Recognizing that most of the science done in the lab happens in the 166 hours each week when we are not together in our lab meeting. This means that big goals are rarely achieved in a single meeting. Meetings are a kind of scientific practice that has important by limited utility. Last quarter, I was hoping for a sampler of DEW delights. I got it.
3. Attempting to patiently lead the group to good science and new insights by keeping the focus on the interesting phenomena and trying to convey why those phenomena are so interesting. We need excitement in our lives and we need to believe that what we are doing is worth doing. I get that by knowing that what we are doing is really new, and has the potential to transform cognitive science.
4. Teaching and learning by doing (DEWing) rather than by talking about doing. Over the years I have learned to be patient here as well. I was entirely satisfied with our brief presentations because I was able to collect from them the lovely list of challenging phenomena with which I opened this note. When you come to my lab meeting I don’t care if you look like what anyone thinks a scientist should look like. I just want you to be doing good science.
That’s my DEW manifesto. I am very excited about the fact that we now have a group of sufficient size and talent to make substantial progress on some of these very difficult issues. We’ve got a critical mass of cognitive ethnographers/ethologists who are not afraid of the real world, not afraid to defy theoretical dogma, and not afraid to use whatever methods are appropriate to develop our scientific understanding of key phenomena in cognitive science. I am looking forward to continuing to work with all of you in the coming year.
Ed Hutchins, 2008