Made out of people

Dissertation draft

... and here it is. This is the same draft as my comittee is going to see, If they want it revised (and I'll know on August 6th), your comments are welcome after that. But if they don't, then this puppy goes out on cotton-bound high quality paper, two copies, and shipped to the library with a pretty signature page.

Dissertation draft in one 214-page file, and four happy megabytes of PDF. (Why is it half the size of the chapter-by-chapter version? I have no idea).

the school of information and computer science
with
the institute for software research
and
the department of informatics
present

Danyel Fisher
in

um, social networks. and, um, time. it's pretty cool

a dissertation defense

foshizzle, my social net fizzle!

seven years in the making

SOCIAL AND TEMPORAL STRUCTURES IN EVERYDAY COLLABORATION

a paul dourish production

free & open to the public
not available in stores

FRIDAY, AUGUST 6, 2004
ICS 2 Room 136
University of California, IRVINE
10 am

An undefended text of the dissertation will be posted by 12:00 am Saturday morning, July 31, 2004, to http://drzaius.ics.uci.edu/blogs/danyelf

Copyright (a rant)

ranting below the fold follows

ACM Copyright says:

ACM aims to serve readers' and authors' interests by publishing high-quality original works, maintaining the integrity of these works, defending authors' rights in them against plagiarism, providing a stable means of linking to them, promoting the dissemination of these works to the widest possible readership in contemporary media, and preserving access to them indefinitely despite changes in technology.

While I don't doubt they are sincere, the problem with locking stuff up on their private server (this goes for JSTOR and IEEE and the various other libraries too) is that sometimes your server goes down.

Right now, ACM is telling me:

Error Diagnostic Information

An error occurred while attempting to establish a connection to the service.

The most likely cause of this problem is that the service is not currently running. You can use the 'Services' Control Panel to verify that the service is running and to restart it if necessary.

Windows NT error number 2 occurred.

Which, while fascinating and--most likely--transient, is NOT what a guy working on his dissertation needs for the four or five references that he's trying to pop into his work at the last moment.

The problem is, see, ACM provides a reliable digital library and prevents anyone (except the author) from keeping a copy around. Which means that when the digital library goes down, we're just stuck. While this is nothing compared to the restrictions of a paper library, which can not only catch on fire but often close in the evenings, it's substantially worse than allowing documents to propagate. Say, to Google and Citeseer, which each have their own reliability problems... but collectively are very powerful and reliable.

Dubious Statistics

So I try not to bring up politics on this blog. There's lots of other wonderful stuff in the world to talk about, like social networks and visualization and temporality research and statistics and ....

Well, let's talk statistics. To do that, I'll need to talk politics for just a moment. Below the fold, perhaps.

Chris Cox is my local US representative. He proudly sends out an Annual Report that discusses government dealings. It's actually useful: I'd never known that having a President costs $365 million a year¹. And it's kind of nice to see the relative sizes of different chunks of the government.

But on the next page, Cox tries to make sure you get his message. And so, using a trick right out of How to Lie With Statistics, he picks some figures that look like what he wants to say.

Looks pretty good! Economic growth is "current"ly higher than it has been in the last few decades!

Until we read the caption.

"Average annual growth in gross domestic product per decade, and for the 12-month period ended March 31, 2004." Why doesn't the chart show 2000-2002? Did those years just not exist? Indeed, given that this decade is less than half over, why pick a decade resolution?

I've gone ahead and prepared a different chart. Let's go to the data. This chart gives us quarterly change in GDP (in normalized dollars). He's not wrong about his numbers--but that doesn't make the chart accurate.

So I went ahead and took one year sliding averages of the percentage growth². It's a pretty noisy graph, so the moving average smooths things out a little. Here's the new variant:

Hm. Looks pretty variable. Economic growth is now climing, but hit a pretty low point recently. It obviously fluctuates a lot.

The other charts do similar things:

Average annual unemployemnt rate per decade, and on June 30, 2004.
...
Average Federal Funds rate per decade, and on June 20, 2004.
...
Average annual rate of increase in the consumer price index per decade, and for the 12-month period ended May 31, 2004.

The figures are, in other words, pretty clearly cherry-picked. This isn't how I want my CEO reporting to me, and this isn't how I want my congressman reporting to me either.

¹ At those rates--a million dollars a day--we could probably outsource the office for considerable savings.

² It occurs to me that the running average of cumulative percentages probably isn't the right measure: growing by 20%, then growing by 10%, isn't an average 15%, it's an average 14%. But that's not what I did here, because I don't want to fighjt with Excel.

Terrible registration experience

So I decide to register my brand new Adobe product.

The label says "You can register your products online at www.adobe.com/support."

I can handle that, and I go to that page. Nowhere to register. (Can you find it?)

I end up dropping into the search engine, and searching for "product registration", which gets me here. That's a good start, although I have no idea how I'd have found it on my own.

(Note: The last of these three issues is already mentioned at This Is Broken)

[more below the fold]

Enter the basic information

So I enter the relevant information (serial number deleted). Unfortunately, there's no choices to enter my product version:

Fine. I hit submit anyway, and get an error message. Fortunately, the error screen now has a couple of choices to hit.

So I select a choice (why this ordering? "7, 6, 5.5, 7.1"?) and now I can move onward.

Almost registered

Sounds like I'm getting somewhere. Just one more form to fill out: my Adobe ID. Here's the form:

I think I used to have one, but I don't remember. That's a pain. Good thing there's a "forgot your password?" button. I'll press it.

Ok, forget it. I give up. They lose.

Update:
Ok, so I didn't completely give up. I put the URL of this entry at the Adobe customer service form which auto-sent me this response:

Thank you for your interest in Adobe product registration.
Register your Adobe product online to receive expert technical support, new product announcements, special offers, and early notice of product upgrades. Products downloaded from the Adobe Store are automatically registered at time of purchase.

For information on finding your serial number, read the Adobe knowledgebase document "Locating your Adobe serial number."
http://www.adobe.com/support/salesdocs/10686.htm

ERDPS: 1090887253.13744.4 (DO NOT DELETE THIS LINE)

I'm not impressed yet.

Update, 7/29/04
bq.. Hello Danyel,

Thank you for contacting Adobe Customer Service.

For your records, your customer ID number is 113606378. The customer ID
number is the easiest way for us to access your account in our database.
In the future, please reference this number when you contact Adobe.

If you would like to register your product online, please visit the
Adobe® Web site at the following URL:

https://www.adobe.com/store/customerregistration/your_account.jhtml

*Note: Either login with your Adobe ID you created, or register for an
Adobe ID.

For further information on registering your Adobe products, please visit
the following customer support page:

http://www.adobe.com/support/salesdocs/217e.htm

Also, visit the following URL on the Adobe Web site for the latest
customer service and technical information:

http://www.adobe.com/support/main.html

The information provided is documented in case #: 1990421.

For more information on Adobe® products or services please visit us at:
http://www.adobe.com or contact Adobe customer services at 1 (800)
833-6687. Customer Service Representatives are available 6:00am-8:00pm
PST, 7 days a week.

Best Regards,

Sarah Y.
Adobe Customer Service

Well, THAT clears it all up.

Small things make a difference, I suppose

Ze Frank writes:

I recently changed the front page of the scribbler - instead of going directly to the app, a screen allows you to branch to the gallery, about, robot, and live vid stuff. the reason i mention it is that such a small change in presentation (not content) has led to a 3 fold increase in traffic to the scribbler. small things make a difference i suppose. i'm going to where my underpants inside out today.
bq. - Saturday, July 3, 2004 09:58 a.m.

[More below the fold]

For a while, I've been thinking about how things we consider small can make a surprising difference. As a techie, for example, I am very aware how trivial it is to make small changes to, say, a user interface: to redraw a line slightly thicker, to put text inside a box rather than outside it. It's easy enough that I often don't think about it--I just put something down and worry about it later.

Turns out that users care. Turns out that other techies care, even if they fervently deny it. This is no surprise to the graphic design community, of course: there's a reason why there can be an entire conference around typefaces -- and, of course, I write this on the eve of the biannual attempt to encourage HCI people and designers talk to each other, DIS.

But I digress. Part of the intent of my research with Martin on multiscale analysis of visualizations (read the paper referenced at bottom) was to try to describe what sort of difference these small things made.

We found some results that were startling to us--even though we both work frequently in this space. His Market Map looked far better with a small tweak to alter the thickness of edges. (This is documented a little bit in the paper).

And here's a redesign that Inxight did of its own hyperbolic tree. The lighter version is a lot easier to read, and figure out what's going on.

The code difference is probably all of ten lines.

After that, my own work got a substantial redesign too.

Dissertations are bad for your brain

Hopefully, exiting grad school will help this a little....

Orkut on eBay

Orkut invites are now being sold on eBay as a way to get to Gmail:

***10 Invitations to Orkut ~ Key to Gmail Invitation***

What on earth would anyone do with 10 Orkut invitations? Even if the idea is that you could join Orkut and talk someone there (a Google employee? a gmail early adopter?) into sending you an invitation, how would the other nine identites help?

Is this a newer, more interesting form of Fakesters? Or just a quick way of making a buck? (The fact that Orkut invites don't seem to be getting bids suggests that this sell isn't working: 20 Orkut invites is going for a penny, as is one GMail account.

Defending Hatester

My advisor, Paul Dourish was just at a conference on mobility-oriented, public social software.

He rants, humorosly, about the state of the software:

My friend Genevieve refers to a large class of mobile and ubicomp applications as "girlfriends for geeks." Basically, these are all the applications that attempt to match you to people you might have shared interests with (including shared friends) as a form of promoting social contact. The underlying premise of all of these of course is that people want to talk to each other (or, more likely, "noone talks to me" or "I don't know what to talk to people about.")

Since I find myself becoming increasingly cranky and anti-social, I find it hard to accept this premise, at least as it applies to me. So instead, I want the opposite. I want a bluetooth app for my phone that detects when there are people around me I wouldn't like, and helps me avoid them. Perhaps it can chart a route for me to escape without running into them. Perhaps it can characterise nearby spaces in terms of how much I'll hate them. Perhaps if it detects that I'm in close proximity to someone I won't like, it'll ring my phone to give me an excuse to get away ("What? Little Johnny's fallen down the well? I'll be right there!")

During the event at Intel Berkeley the other day, after a series of girlfriends-for-geeks presentations, I made a note to myself about Hatester.com, a site where you could describe who you dislike, and then find out the most disliked people, chains of dislike, degrees of reciprocity, etc. The marketing slogan would be "because the enemy of my enemy is my friend." It turns out that the domain is registered, even though there's nothing there. Oh, but I see that there is something at hatester.org....

Of course, the dislike network is extremely difficult to figure out. I suppose transitivity is "the enemy of my enemy is my enemy," and reciprocity means "I am my enemy's enemy." But chains of dislike?

The Social Computing Research Lab List

Even though many of the great old research firms may not be what they once were (PARC has lost its Xerox moniker; AT&T has lost much of its old social computing staff), industry-aligned and sponsored social computer research is far from dead.

This has come up in the context of my job search I'm leaning toward industry, largely because the sort of things that I do seem to connect very well to current industrial interests in a number of different ways. Industry seems to be the place today where research can have short- or medium-term effects on product. (Of course, there's still a difficult balance of how to get those products into the real world, a venture with mixed results)

I've run into quite a few places that are doing interesting things. Here's a brief overview of labs you might want to know about: places whose publications are worth monitoring.

1. Microsoft maintains a community technologies group, a social computing group, and an adaptive systems group.
2. IBM has labs in Cambridge working on collaboration, Almaden working on user experience, and New York
3. Intel's "lablets" have a People and Practices group, as well as teams interested in social computing at both Seattle and Berkeley
4. FXPAL is an American presence of Fuji-Xerox that does research in Palo Alto
5. Hewlett-Packard's information dynamics lab has interesting work on social networks and social structure
6. PARC has projects in Sensemaking and Community
7. MERL is an American presence for Mitsubishi in Cambridge
8. MITRE is a government-oriented research organization with interests in collaboration and information solutions
9. AT&T has a strong Information visualization group. (It is a peculiar habit of mine to think that info viz is a form of social computing.)
10. Google seems to maintain a research staff in a variety of cool stuff, but they don't talk about it much. (Am I just missing a major Google publishing venue?) Still, anyone who would bring in danah as an intern knows where to be looking.
11. NASA has research on human centered computing, including some social and groupware research
12. Nokia, Boeing, NIST, and Sun also have various forms of research in these areas...

Update: Know of anything I'm missing? Please let me know.
Update: Added Boeing and PARC.
Update 7/20: Corrected language about Intel

Will trade food for comments.

Update: Links removed. See newer version

This is it in first draft form: most of the words, none of the prettiness.

7 chapters plus an incomplete technical appendix. 5 zipped megabytes. 4 different roman body typefaces, at last count. Microsoft Word. (Word count below the fold).

(By request, 8 happy megabytes of PDF)

Next week is entirely dedicated to making this thing look good, have a correct bibliography, and so on.

Want to give it a read? Feedback eagerly sought until July 30th. (After July 30th, and before August 6th, I'm officially going nuts worrying about my defense. Feedback eagerly sought after August 6th, but there'll be a better draft then.)

In exchange for your thoughts, I'll take you out to dinner when I next see you.

I'm particularly interested in the thoughts--even casual ones--by anyone who hasn't been immersed in this stuff for years. Does it get too rapidly geeky? Do I flit like a butterfly between important concepts, especially when I might linger more lovingly on something? (This is most likely a problem in the conclusion).

The word count:

Chapter 1: 4500 words (17 pages)
Chapter 2: 7800 words (23 pages)
Chapter 3: 4200 words (22 pages)
Chapter 4: 6700 words (41 pages)
Chapter 5: 3550 words (18 pages)
Chapter 6: 2700 words (13 pages)
Chapter 7: 1300 words (7 pages)

My dissertation, in number of pages, appears to follow a rough Poisson distribution. I'm not sure how to interpret this. Number of pages is only weakly correlated with number of words. Mysterious.

Typeface and the Single Dissertation

Ok, I'm sick of staring at page after page of Times New Roman. Which is why my dissertation is now in a variety of fonts: Chapters 4, 5, and 6 in Georgia; Chapter 7 in Century; Chapter 8 in Book Antiqua.

What would you set a dissertation in?

Naming Conventions

In which the author tearfully confesses a series of abysmal research project naming conventions, and realizes he has managed nearly all of them in his dissertation work:

Updated to correct date typo

1. Encyclopedia Brown That is, a project is named after clever-if-obscure reference. Thus, for example, my dissertation's infrastructure is named "Soylent" (because it's made out of people). Those who get it may or may not enjoy the extra layer of reference to the server name for this blog (DrZaius).

1. BicapitalizedNaming Remember when everything was about MungedTogether names? I need not actually name names, but I will admit that my dissertation's first-ever application is called "TellMeAbout."

1. SUI2C (Silly Unpronouncable Initials To Confuse). GNU may not be Unix, and YACC may be yet another compiler compiler, but "Bison" is an obscure reference (see above). "EE4P" is Enhanced Email for People; after a team of undergrads had played with the system enough, it stuck.

1. MASIT (Meaningless Acronyms Spell Irrelevant Terms). I once was involved in a project named EUPHORIA (End User Production of grapHical interfaces fOr Really Interactive distributed Applications). I'm now on an open source project, JUNG, the "Java Unified Network/Graph infrastructure". The title--containing a vague reference to a pychologist--is awfully hard to find in a Google search.

I think that's my own list of sins--how's your naming convention doing?

Update Auros reminds me of one important additional type:

1. Strawberry2Blueberry As far as I can tell, most software spends most of its time converting one thing to another: a graph to a list, unicode to ascii, Java to Fortran, whatever. While I'm fairly sure that I haven't committed that particular sin much, it's certainly still with us in all sorts of code.

The Coming Campus Water Crisis

Update 7/16: Jack spent a few hours on the phone yesterday. Never did reach the campus rep, but he found the main office, who--on the second call--was able to look up our contract (an impressive feat: the only unique ID--besides the account number--they keep is the phone number, and all of UCI uses the same phone number. Worse, we never got our new account number after we transferred from the Sparkletts).

We got 7 of our 11 scheduled bottles, which is a pretty good start.

---

Our school used to have a contract with Sparkletts water. They brought us a eight or so five-gallon bottles every few weeks for the hallway, and there was water. We were very happy with all of this, and it kept us well hydrated. (Bottled water is a good idea here; for some reason, the tap water smells a little off.)

Recently, the school changed over to an Arrowhead contract: I believe it was a cost savings of some sort. All the Sparkletts bottles were taken away, and we got one bottle of Arrowhead, which we drank dry in about two days. We can't get in touch with the Arrowhead representative (Gary Amador), or Arrowhead support, or any other people. Nor can at least one other part of the department that changed over: the grad student contract, the office contract, none of them. Arrowhead doesn't love us.

So: Sparkletts good, Arrowhead bad.

Just to let you know.

[This entry provided for your Googling pleasure. It will be changed when water shows up.]

Dissertation progress

Well, I don't have a thesisometer on my site. (Maybe it's a lack of competition?)

But I do have one on my door that shows drafts of various chapters as they come together. And I'm excited to announce that my dissertation will soon be available on this very blog. In particular:

• My dissertation's "reading draft" will be posted this Friday. This is without final gloss editing, but should have most of the words in place.
• My dissertation's final draft will be posted two weeks hence, on Friday July 30th.
• My dissertation will be defended on Friday, August 6th at 10 o'clock in the morning, in ICS-2 room 136. Feel free to drop by!

ClearType and Rotating Monitors

Ok, so I love ClearType: it really does make my monitor look prettier. Enough prettier, I'm afraid, that I really don't like having it off. Which means that my nicely rotating monitor, perfect for displaying a full sheet of text, never gets rotated. Because when I rotate it, I need to turn off ClearType. (CT depends on attributes of the ways that pixels are physically aligned on the screen.)

Any solutions? Is there a perpendicular ClearType driver out there somewhere?

Does Bill Tozier have it right?

Bill Tozier, eBay observer, Erdos-seller, and diverse thinker, wants to create a new scientific community. One in which papers can be openly presented, hashed to pieces, and the best of them can be extracted and the strongest ideas revised. In the article linked, he discusses how to deal with the various crackpots and loonies who would inevitably become involved in the space.

The short summary is that a combination of rules and interest by the contributors. The contributors who have failed to take into account a substrantial body of literature should be directed to it (where they can learn how to use vocabulary well and understand what's already been done); the space should have rules about standards of typography, language, and presentation (to avoid the hand-scrawled diagrams with random capitalization).

These seem like perfectly reasonable standards for the sort of online, multi-edited journal that has been proposed for years. Indeed, arXiv.org seems to be slowly evolving into something like this: it's changed from pre-prints to "e-Prints"; some articles are printed in it that appear nowhere else. (Why they appear nowhere else is a matter of specuilation left to the reader).

My concern with Tozier's system is the critical mass problem. I suspect that mad geniuses will be delighted to use it, once they are convinced that the inability to use the blink tag doesn't actually damage their content. (That is, assuming that the contingent wants to be published in well-read sources.)

But what about everybody else? Will there be a large enough population to wade through the chaff and find the wheat? When reviewing for a major conference this year, I found a paper that flagrantly disregarded twenty years of research: not that it drew opposite conclusions, but that it simply failed to note that anyone had thought about the topic before, and wrote itself as a sort of prologomena to the field. Just getting one of them made me rather irritated at the authors; what would have been the effect of dozens of those?

For me, I would have stopped reading. And gone and swam in a pool that wasn't so polluted.

How do you ensure that it's worth the commentator's time to participate in a community? How do you make sure the community is big enough to provide value? CSCW research has lots of studies of communities that failed, didn't really take off, or sputtered. While Preece's book tries to engage this issue, I don't think there are yet canonical bullet points for "making sure your community gets adopted."

I'm curious to see where Bill's work goes with this.

Autocollaboration

Our term of the day is one that I made up during a job interview. I was trying to explain what categorization problems (such as those made visible in del.icio.us) and described in Sorting Things Out) have to do with CSCW.

One of the major things that one does with categorization, of course, is put stuff somewhere so one can find it again. You want others to find it too, of course. But sometimes, it's just about finding it yourself some other time. That's a form of collaboration, of course: it's collaboration across time, instead of space. And it's collaboration with yourself.

"Autocollaboration."

I like it. More later, I think.

Games and Sound II

In my last entry, I talked about a variety of game sounds, and discussed some of the technologies for adaptive music systems linked to sound. In this entry, I’d like to spend a few minutes on some of the ideas that I see growing from that.

In my last discussion, I emphasized “subtle”. Here’s part of why. My friend had talked about cartoon-type sound tracks: when you walk up the stairs, for example, the game would play “stair climbing” music, perhaps with each successive note getting higher: “plunk plonk plenk plink!” When you walked down, it would play it getting lower. Users want power over their games, and many try to explore every inch of the game world, and—wow! That’s a mini-game. What happens when I walk half-way up, then turn around? And jump off? And then walk up again? The user, now actively playing the game of “learn about the sound system,” has forgotten about the underlying game of “shoot the enemy,” and is now wandering around the sound system.

Presumably, that's not realy a game goal, or a direction the designer is trying to go. And it's a distraction, rather than an enhancement, to the game.

So let’s grant that this is a cartoon world. Give me the goal of generating Carl Stalling¹ music: a good solid trombone “thwaap!” when you throw a pie; a “sneaky” theme when you get on tip-toes, and so on. Carl Stalling, however, had an advantage that no game player ever will: he knew what was happening next. And as such, a lot of his music anticipates the next step: the trombone subtly sneaks out of the theme so that it can be ready with the surprising “thwaap!”; the touch of cymbal anticipates the sound of sneaking around the corner, and so readies you for the rimshot when you are discovered.

But when a character is wandering around the world, you can’t be sure that he will be compliant. How do you ensure that the pie will be thrown on beat? How can you know whether the AI will catch him (and so the sneaking / rimshot combination is correct), or whether he’ll make it past (and so a different music set might be more rewarding)? In other words, a movie composer can see into the future, and a game designer can’t.

There are certainly clever solutions--I talked yesterday about a skating game that melded into a (beat-less) riff at the beginning of a jump, and so could hit a downstroke when the board touched ground. I also got a comment on the last article, discussing a game with different zones, and a bunch of bridges between them. As the player moved between zones, the different bridges would smoothly play. Which meant that the music would shift seamlessly as the player wandered about the world.

But can we do better?

A number of years ago, I saw a dynamic adaptive music system out of Microsoft Research. I don’t recall the name offhand, but it was a research project that allowed a user to start with a basic musical theme. It would then produce simple variations on that theme based on notions like “louder” or “faster” or even “more exciting” and “funkier.” As you played the music, you could order it to get solemn, and the system would smoothly transition to playing solemnly.

So in my mind, an adaptive sound system could use that sort of idea to generate a system that pays attention to the user’s state. Like Splinter Cell, it might play different music for “seen” or “visible”—but it might do better than that. Let’s look at a few dimensions that really are available to the system. Let’s pick a simple first-person shooter with a stealth component (say, “Counterstrike” or something like that).

• How vigorously is the character moving? He is now running / jumping / walking / crouching / hiding against a wall

• What’s his military stance? He is unarmed / ready with a tool / armed with a small sneaky knife / has a shotgun in one hand

• How’s he doing? powerful / ok / wounded / hopped up on steroids / about to die

• Does anyone see him? a guard just noticed / alarms have been sounded / no, he's in the clear

You can even add an aspect of judging the gameplay:

• He’s got a knife? He is sneaking up behind a guard / he is insanely running into a machine gun nest.

• He’s got a shotgun? good way to kill a zombie / useless against a horde of soldiers

• Is he running into battle / away from battle / covering great distance with no enemies near.

And so on. I think you can probably handle this combination of stuff with both ambient music and event sounds. And there might be more than one theme to be explored with its variations. But I think that setting up a couple of axes to the system could then allow dynamic, unpredictable music to be generated. The composer lays down the theme and works out how its variations will be played—but the game is responsible for putting the pieces together to decide when to play “sneaky” or “sneaky with spotted” and even “sneaky, and sneaking up behind the bad guy.”

My suggestion, then, is that with a system that knows something about how to score different sorts of music--on a subtler level than "louder" or "softer"--you might be able to do this with fewer than the nasty-seeming combinatoric matrix of sounds. Start with a few themes, adaptively twist them for different moods...

Just a thought.

--
I should point out that none of this is new: again, I'm a non-expert. A google search for "adaptive music" finds hundreds of hits; Gamasutra has an interesting article from 2001 about adaptive music, which touches on some of these points..

Indeed, I just noticed Adaptive Audio that engages some of these issues, including two very interesting case studies.

Games and Sound

A few days ago, I had a conversation with a friend who is deeply involved in the music design business for video games. After an undergrad degree in movie music composition, he got involved in the core design team for a major game sound platform.

This is a two-part blog entry. In this, I’ll talk a little about the unfortunate state of the game music industry—as summarized to a non-expert, over a beer or two, after a long day of doing other things. Again, I'm NOT AN EXPERT: I don't even play a lot of games. Rather, a conversation has stirred thoughts, and I want to talk about them.

In the next, I’ll blue-sky for a few little bit about possible designs for more adaptive, dynamic music in game systems. Neither, I’m afraid, will be explicitly about anything being made out of people: however, this entry is about the social process involved in game design; the next entry is about the use of creative design for sound systems, based loosely on some of my various work.

My friend was discussing his work as a designer and as a consultant. He’s the program manager for the system, but spends a lot of time going to trade shows and game companies, evangelizing the facilities that his system provides, discussing the technical aspects of it, and exploring different ways that it can be used. His API essentially makes it easier to connect game events to sounds, and thus allows the composer’s job to be less programming-intensive.

This helps with some of the basic issues: it provides a vocabulary for wiring game events to sounds, so that it’s easier to code “the enemy fired a bullet, forward and to the right”. (*Update*: In case I wasn't clear before, this is pretty classic. In lots of games, audio is a crucial gameplay aspect in figuring out what's going on, is is a useful second cue. I'm more thinking about whether audio can be more sophisticated than the fairly realist take of "I step, and it goes 'thock.'")

Unfortunately, game audio is too often an afterthought. He complained about game designers who bring in music “because it’s missing something; someone has to make it fun,” who hire composers after the game is nearly finished, or who just ship out the sound design to a couple of interns.

Thus, in a world where the game developer has the resources to locate 3-D sound anywhere around the player, who can simulate curtains and echoing walls and footsteps, who can generate music on the fly, the designers often settle for a loop of music in the background and a line of recorded sounds in the foreground. This makes for a fairly dull experience, when sound might conceivably lighten the game up and add a great deal of subtle excitement.

Subtle, of course, is the critical word. It’s easy to see a good graphics system, and it’s fairly quick to observe good game AI. But what does good sound sound like? “Well, it’s CD quality” doesn’t cut it: is it exciting? Is it fun? Does it build on the game experience? Unfortunately, humans are far more sensitive to—or at least have better vocabulary for describing—video than audio, and so the music is often under-evaluated. Press kits include videos and pictures, but never game sounds without pictures. While game soundtracks are becoming popular, that’s more of a celebrity music track than it is music as part of the game. (My friend pointed out that many game reviews never bother to mention the sound at all, or limit themselves to words like “realistic” and “amusing”.)

The unused potential is nearly infinite. Think of a character walking behind a curtain ringing a bell. As the player moves around, a good ray-trace should be able to determine whether the bell sounds local or distant, muffled or loud. Unfortunately, that’s a lot of work—and rather computationally expensive. Yes, some of that information is already available to the game engine: after all, it’s doing some ray-tracing already to figure out where the curtain is, and whether the player can see it. But the graphics-oriented ray tracing system is oriented towards things that are visible, and “behind a curtain, and behind the player” isn’t quite there. (I have seen an old SIGGRAPH paper on ray tracing sound, but no matter how you see it, the system gets expensive: is that payoff to do so much work worth it when you can approximate? “In this room, play the sound quietly; in that room, play it loud”.)

Of course, the bleak picture I paint above isn’t the only thing that’s happening today.

First, music oriented games think hard about music. Amplitude dynamically mixes tracks, and adds other audio cues both to direct the game-play and to provide feedback on success, Dance Dance Revolution uses music to guide the player’s steps (to some extent—I’ll have to blog about that some other time), and uses crowd applause to reward skillful play. Space Channel 5 synchronizes the game around music.

Second, one can do moderately-more sophisticated variants of the linear play. For example, the recorded samples in a game might have slight variants: a footstep might sound randomly slightly different; a gun cartridge casing might eject with either a “plink” or a “thunk.” The music might vary level by level, or even region by region. And ambient music can be played out in different ways. For example, many games have an “outdoor ambient” track: it’s the birds chirping, the crickets humming, and the wind blowing. While the game may use up to a minute or so of this track, they often simply repeat the track over and over. Users begin to notice this, and start to get bored: “ah, the crickets have chirped. Next, we’ll hear some wind.”

In contrast, Microsoft’s “Halo” works slightly differently. The same minute has been recorded, but now it’s sliced into five-to-ten second segments. Those segments are played back in a largely random order, which creates a constant sense of novelty.

Last, one can synchronize the music with the action to create a more integrated experience. It’s this, actually, that I’m most interested in, and will be discussing more in my next entry. But to start off, there’s a couple of variants that are worth noting: one is the ways of cueing different sorts of music to the action. For example, “Splinter Cell” plays different music when enemies have spotted you and are actively hunting, as opposed to when you are unspotted. (It also plays louder footsteps for running than walking, and so gives some feedback to how audible you might be to an opponent).

My friend also discussed a snowboarding game, with DJ-mixed music in the background. When the player did a trick, the system always made sure that there was a solid downbeat at the moment the snowboard hit the snow again—which added a bit of emphasis to the jumps. (Turns out the system also played higher-pitched, quieter music during the jump, which helped ensure there was greater contrast upon landing.)

I think that’s a decent non-expert, birds-eye taxonomy of game sound. In my next entry, I’ll discuss some of my thoughts in those last categories: game music linked to game play.

Conferences

Am I the only person who sees a conflict with having SIGGRAPH (August 8-12, LA) and ASA (August 13-17, SF) so very close together? I'm not sure I'll be able to attend both, and defend my dissertation, and do revisions to my dissertation, and still squeeze it all into August.

(What? You mean some sociologists aren't interested in computer graphics, and vice versa? How confusing!)