one question implicitly raised by the Barkhuus and Dourish paper is that of who does the analysis of a deployed system. on the one hand, you can have the the system architect/builder/designer/implementer perform the analysis. on the other hand, as is the case in this paper, you can bring in someone separate from the design team to perform the analysis. is either one necessarily better? what are the situations in which one is preferrable to the other?
I suspect that neither is necessarily always better than the other. furthermore, I suspect that there are not any general situations where one is preferable to the other. to be sure, each approach has its own strengths and weaknesses. choosing the approach is a matter of deciding why you are doing your analysis and what you want to get out of it.
when I was at the AAAI conference last summer, there was a group from CMU doing studies of a robot demonstration they had there. rather than doing the evaluation themselves, however, the group had brought on an anthropology student to study the way that people interacted with the robot. as I understand it, there's actually a large push for this study of HRI (human-robot interaction) with an anthro-/socio- emphasis at CMU. bringing others like this can be really beneficial, as outsiders won't have the same experiences with the technology as those who built it, having to explain the system to others can help the designer concretize their thinking, and those not involved in the creation of the system will bring in different biases than the creators.
however, this approach does have its problems. such non-CS types are not always available to do the analysis. furthermore, when they are, it takes time and effort to bring them up to speed on the system, describe to them what the intended locus of study is, etc. it also takes time for them to translate their analysis back into lingo that can be understood by the computer scientists. there's been some work done on the role of ethnography in system design (Hughes and Anderson come to mind), arguing variously that ethnography should not just be in the service of design. that's not really what I want to talk about it, but it's certainly related.
so, (someone clever apparently said), if getting ethnographers to do analysis for our systems can be so difficult, why don't we just train our computer scientists to do ethnography? indeed, here at UCI, there's a qualitative methods class, generally taught by the professor of this class, that attempts to give CS types an introduction to some anthropological and ethnographic methods. in general, this is great; bringing different perspectives to your work usually opens your eyes to seeing things in a new way and opens your mind to new insights. however, I think that a lot of CS people also realize that ethnography is really hard, in-depth, time-consuming work. it's already hard enough to design, implement, debug, and deploy the system, says the computer scientist, and now you want me to do all this ethnography, too? in trying to avoid all the tedium of ethnographic analysis, we end up with systems implementers doing "lightweight" ethnography, such as (no disparaging intended) Paulos and Jennings' urban probes, or people just doing qualitative observations and calling it ethnography. in certain situations, these methods, can be interesting, provocative, and occasionally lead to important topics for study and discussion. however, I believe that the majority of the time, they are used in place of more time-consuming, rigorous methods for qualitative, anthropological, or ethnographic study.
there's also a note to be made here about corporate anthropology (e.g., Intel's peoples and practices research group). however, (a) I don't know enough about this area, and (b) this post is already long enough.
so why does there seem to be this big push to turn computer scientists into sociologists, anthropologists, and ethnographers? I understand trying to bring new perspectives, and I definitely understand trying to give an appreciation for other disciplines' methods. however, why not, as is done here, bring in others to do the analysis of your system rather than trying to do it yourself?
as a side note, there's also the question of authorship. if I build a system and someone else does the analysis, who gets to be author on the paper? if it's a paper about the system with an evaluation section, do I credit the analyst I brought in? if it's a paper about the analysis of the system, does the analyst credit me? here, it seems that authorship goes to the analysts, and the system builders get a bunch of citations and acknowledgements, but I wonder if it could have gone down differently.