The dataset consists of conversations between dyads that took place while they completed two different interactive tasks. The conversations were recorded, transcribed, and segmented into utterances according to some criteria. This means that there are repeated utterances from each participant, and from each dyad. Different research areas use different terms to refer to this kind of setup: repeated measures, panel data, clustered data, etc. The analysis is further complicated by the fact that the predictors and variables are all categorical. Some are binary, the presence or absence of something. The more interesting variables have more than two categories (in some cases, MANY more).

I am trying to estimate the strength with which each of a set of 15+ utterance goals is associated with one of three roles participants assumed as part of the study. To do this, I need to specify a mixed-effects multinomial logit model, with a set of fixed-effects categorical predictors and a hierarchical random effects control for participant within dyad. This involves choosing a reference category of the response variable, and then running a series of binomial logit models that compare all the other levels of the response variable in turn with the reference category.

Here is where I am running into a situation, again, where I am pushing up against what mainstream statistical software packages are reliably capable of, and even R does not seem to be able to do what I want without more programming than my meager statistical background has prepared me for. The problem as I understand it is, each one of the binomial logit models that makes up the multinomial results uses a different subset of the data, excluding those observations that are related to the levels of the response variable not included in the model. This means that the random effects are estimated differently for each binomial logit model, depending on which observations are included in the subset. The upshot of all of this is the overall multinomial model estimates come out differently, depending substantially on which category is chosen as the reference category.

So that’s the problem. However, I did not write this to whine about how I am stuck. I’ve been trying to figure out a solution that I can live with… do I bail completely? Hire a real statistician? How can I figure out how biased the results would be if I were to to do a purely fixed-effects model? (Without random effects controls, any results produced might in fact be due to some unique aspect of the conversation within a particular dyad in a particular role, rather than indicative of something that shows up across all of the dyads.)

Researchers in many fields work with categorical data, and at least some of them over the years must have encountered this problem, whether they knew it or not, and were faced with the same tradeoffs. In order to get the paper out the door they had to just pick a compromise and go with it. But, any results reached due to a compromise are biased in some way. Models like this are just now becoming possible for people like me, with just enough stats knowledge to be dangerous, to run using fairly standard statistical software packages. But what about all the research that has come before — how accurate are those models, and the results they produced? How much do people allow what is statistically feasible to determine their research design, vs. compromising on the analysis after the fact? We all stand on the shoulders of giants, but how often were the giants using naive or incorrect statistics?

I recently came across mention of an alleged “censorship” controversy related to Digg.com [see here, and here for mentions in mainstream media], in which a group of coordinated users apparently succeeded in preventing certain stories they found politically objectionable from reaching the front page of Digg, so these stories would not receive wide exposure. The users achieved this end through what is essentially a thumbs up / thumbs down mechanism fundamental to the way Digg works, by which users vote on whether stories should be promoted or buried. As one blogger points out, opinions about censorship aside, these users were operating within Digg’s available functionality and did not necessarily violate any rules. People who are upset about this use of Digg’s voting mechanism claim the group of users were gaming the system — coordinating which stories to target via another social media application (Yahoo! Groups).

The “gaming the system” and “censorship” aspects of this controversy are less interesting to me personally, than the flexibility of such a simple voting mechanism, used to express an entire political agenda rather than individual, personal preferences. This is an instance of the point I was trying to make (badly?) a few days ago — that even simple mechanisms can be tools for expressing a wide variety of meaning, but that meaning is not obvious from single contributions. In this case, the coordinated intentions of the group only became apparent in aggregate, and only after people who were pissed off about having their stories consistently targeted for “burial” were motivated enough to figure out what was going on. In other words, the meaning behind these actions was not present in the aggregate voting data; was only visible if you already knew where to look.

In the last year of so of grad school, several of my fellow students at a similar stage in the program started doing “paper swaps” before a big deadline. This was an awesome idea brought to us by @jennthom. Each person who was submitting a paper agreed to review at least one other paper, in exchange for feedback on their own paper. This brilliant plan had many benefits: it encouraged each of us to finish things a *little* bit earlier than we would have otherwise, we got to learn more about what our colleagues were working on, and of course we both received feedback on our own papers and got to practice giving feedback to others. The main drawback was that it created more work at an already busy time.

An added benefit not obvious at first was that when it came time to write rebuttals to reviews for submitted papers, we had a group of people who were familiar enough with the papers in question that we could read each others’ reviews and make suggestions for the rebuttals. The great thing about this group of people was that it seemed like nobody was overly sensitive about sharing their reviews — and I think that this was a great learning too for all of us.

I have two questions based on this reflection about paper swaps and sharing reviews, and I’d love feedback if anybody happens to notice this post and wants to share:

1. How do I get something like this started at a new institution? I think what we did in grad school worked because we were a fairly small group who both trusted each other to be helpful, and were in serious need of feedback. I certainly learned a LOT from the experience, and think it would be super valuable for other students to participate in something similar. But how do I convince people the extra work is worth it, and that there is nothing to fear from sharing reviews? To that end, I am perfectly willing to share my own reviews on both accepted and rejected papers, which brings me to my next question…

2. Is it appropriate to share publicly, like on the Internets, reviews for one’s own papers? Would it just be too confusing for people if there were multiple versions of a paper, or even papers that never ended up being published, available on an author’s website along with the accepted papers (even if there were a separate page for them or something)? Would anyone even be interested in seeing these things? Also, do reviewers expect that what they write will be held in confidence? Personally, I always write reviews (and everything else for that matter) as if I am writing for an unknown, public audience — it is so easy to share these things, you never know who might see them. And I don’t want to say anything in a review that I would be unwilling to say to someone in person. I just have no idea how others feel about this.

I have to admit, I don’t really “get” Q&A sites. Who are these people that ask questions like, “what is $16 and $8.50 american become in canadian?“, “why do people believe biggie is better than tupac?” and “My turtle has broken his hand? Please Help!!!?” — all from the front page of Yahoo answers. Why do people seem to believe they will get informative, useful answers from random folks on the Internet? Do very many people receive satisfactory answers this way? I know that when Yahoo! Answers appear my search results, they are never helpful for me.

One might argue that Facebook users are already asking and answering questions, via the status updates and comments that are already supported. So what’s the point of “Facebook Questions”? I think there are two:

- By choosing to post a question in “Facebook Questions” rather than as a status update, users are essentially adding metadata to what otherwise would be a status update post, informing Facebook that the contents of this post are a question or an answer. If the question had been asked as a normal status update post, it would be very hard for Facebook to automatically determine whether a status update was in fact a question or an answer. Marking something as a question or an answer makes the information that much more useable for data mining and search.

- Because posts to “Facebook Questions” are public by default (unlike status update posts which can be protected), Facebook has invented a way to circumvent privacy controls for a certain class of posts, allowing them to build up a corpus that could generate more ad revenue, and might even be data others would pay to use.

The question I have, then, is this. It seems pretty clear why Facebook would want people to use “Facebook Questions”. But why would Facebook’s users choose to post their questions to a bunch of strangers this way, rather than doing what they are already doing — posing questions to their friends via their status updates? I guess “if you build it they will come” has pretty much been true for Facebook so far… but it is hard for me to imagine what would motivate people to change their behavior in this way. What’s in it for them?

For example, consider Facebook’s “Like” button for the web. Users can contribute metadata to some web pages by clicking a “Like” button that appears on the page. But what did all these users really *mean* by the contribution? Does clicking the “Like” button represent an endorsement of the content? Support for the author of the content or the content provider? Is it a straightforward or sarcastic “Like”? Is it intended to express a sincere endorsement, or was the contribution motivated by some external incentive? How do other users interpret the fact that one page has 300 “Likes” and another only has 3? How should I interpret it? What does that mean for the next web page I visit with a “Like” button?

There is a cumulative aspect to social media systems that makes them very difficult to design and to study. Right now, one has to actually BUILD the “Like” button to find out how it will be used in practice, because today’s utility or benefit derived from participating depends upon contributions by yesterday’s users, and tomorrow’s contributions are shaped by today’s experiences.

This is a big challenge to the traditional HCI development cycle. Think about designing a spreadsheet or word processor application — the functionality is the same every time. It doesn’t change based on who uses it. Not so with social media systems. The contributions of others are an essential component of the system — PEOPLE, and the data they generate, are a part of the infrastructure of these systems in fundamentally different ways from most other kinds of computing systems. Think about designing a new traffic flow paradigm, vs. a new heads-up display for a car. Driving is at once an isolating and inherently social activity. If an individual driver doesn’t obey the rules, both those codified into law and the norms that have developed, consequences spread well beyond him or her.

The people, and their choices, and the traces left behind by their choices (ever been stuck in a gapers block or delayed by rubbernecking on the highway?) are part of the infrastructure. I argue that users’ contributions are as important to the social media infrastructure as the application features and internet protocols and mobile devices and wireless spectrum are. So design requirements for social media systems are really “enabling technologies” for experiences; the Facebook status update is both a feature and an enabler of the future.

I believe the near miss I witnessed in traffic resulted from the existence of invisible infrastructure, developed over time from social patterns. The “zoom around stopped cars to make the left turn arrow” behavior happens all the time at this intersection — so often that when I am driving from the opposite direction, I now leave room for the idiots who think it is ok to suddenly swerve into the oncoming lane. Which of course makes it easier for them to behave this way. This is an informal social practice at this particular intersection — enabled by the fact that the street is wide enough to accommodate an extra lane of traffic, but perpetuated by the choices made by both the the drivers who choose to zoom ahead and the drivers who choose to get out of the way and allow them room to do so. When I first moved here, I didn’t know about this practice and was quite surprised to see oncoming traffic driving straight at me in what I thought was my lane. I was obeying the physical signals — the lines painted on the street to divide the lanes; I did not yet have the same context as the other drivers, which they had built up based on repeated experiences over time.

I am not a traffic psychologist or a transportation engineer. I am sure there are ways to redesign the intersection, or patrol it more carefully, or in some other way make it safer for both motorists and pedestrians. However, I *am* somebody who has a background in Human Computer Interaction and who has spent quite a lot of time wondering what the invisible patterns of behavior look like in social media, and how we might detect, learn from and use these patterns to help users make better choices about what to share, and find the information they need.

For example, I did not adapt to the invisible pattern of behavior at the intersection until I had experienced repeated visits in which I observed and was surprised by the behavior of others. Social media does not provide the information necessary for this kind of adaptation very well:

1. People do visit social media sites repeatedly. But rarely do two different people have exactly the same experience, and it usually isn’t possible for one person to know how much their experience coincides with another person’s experience. For example, Facebook doesn’t look the same to everybody; in fact, because few if any people share the same set of FB friends, no two people experience the very same thing when they visit Facebook. My news feed is unique to me, just as yours is to you. In this way, social media is strangely isolating in ways that we don’t readily perceive.

2. While social media applications make SO MUCH information contributed by SO MANY people available ALL THE TIME, they do a pretty terrible job of organizing and distilling and presenting the information in such a way that it might be possible to see the hidden patterns. We cannot see and interpret signals that might help us theorize and understand what influenced the content or format or timing of a post or its consequences. Nor can we easily understand raw information about how the aggregate of posts changes over time. We see what other social media users choose to contribute or post; but we don’t always see the pattern of reactions or consequences that branch out from a post. It was easy for me to see the consequences of the pedestrian’s expectations about the traffic pattern at the intersection — he was nearly run over. But it requires a third party, like the news media, to convey information about potential consequences to posting what might be perceived as a threatening status update to Twitter.

In Part III, I talk about the cumulative aspect of social media systems, and how the users are part of the infrastructure.

On my drive to campus last week, I witnessed a pedestrian nearly get run over in the street, complete with squealing, smoking tires and a frantic leap to get out of the way. It is hard to say whose fault it was. I was waiting at a red light, maybe 5 or 6 cars back from the intersection. Interestingly, there was no honking involved.

The street I was on is a two-lane street with a short left turn lane, only big enough for maybe 2 cars to wait for the light to change. The pedestrian was crossing the street, but not at the crosswalk — he decided to walk out into the stopped traffic right in front of the giant SUV I was driving, which I am sure he could not see around. He was wearing a bright orange shirt. He paused as he was passing my car to look to his right, making sure no traffic was approaching from the other direction, before entering the other lane of traffic.

Unfortunately for the pedestrian, unique and invisible traffic rules prevail at this particular intersection. Despite the fact that the left turn lane is quite short, the street is wide enough to potentially accommodate a longer left turn lane that stretches farther back from the intersection. So despite the fact that this a two-lane street, people wanting to turn left often have enough room to create an informal, longer turn lane. I regularly see people pull out into oncoming traffic and speed toward the intersection, passing the waiting cars, in order to make the left turn arrow. This is what happened last week. The man looked to his right, saw no one coming, and proceeded ahead without noticing the car speeding up from his left that shouldn’t have been there in the first place. He didn’t even look to his left at all; I presume he was thinking he had already crossed the lane of traffic coming from that direction.

Fortunately, the car’s brakes worked, and the man was nimble enough to jump out of the way. The driver looked pretty freaked out — I know I was. The pedestrian got into his car which was parked on the other side of the street. Then the light changed and I continued on my way.

In the forthcoming Part II, I answer the question, what does this all have to do with social media?

A URL shortener is an online service that assigns a short URL to a web page’s original address, such that the web page can be accessed either by the original URL or by the new, shorter URL. People use these services in conjunction with Twitter, when they’re including a URL in a tweet. Because tweets can only be 140 characters long, a shorter URL means more characters left over for saying other stuff.

When I say there is a plethora of URL shorteners available, I truly do mean it in the “extreme excess” sense of the word. For example, there’s a Flickr set of screen captures that contains images of the home pages of 129 different link shorteners, including shadyurl.com for when you want your “shortened” link to be suspicious and frightening. Bizarre.

Speaking of shady URLs, one issue I have with URL shorteners is that I am never sure what I am going to get when I click on a link from a service like bit.ly, which is the “default” URL shortener of Twitter. Phishing attacks in social network applications (like Facebook and Twitter) are becoming more common, and tricking people into clicking on links that execute code intended to steal passwords, etc., is often the goal.  As a result, I rarely ever follow links that show up in my Twitter feed, or in my email. It just seems to risky to me to click on links from URL shorteners that disguise the ultimate destination.

Interestingly, the distribution of Twitter posts using different URL shorteners has shifted quite a bit over time, as described by a blogger on TechCrunch.com in “What happened to bit.ly’s market share?” The article describes a new “pro” service offered by bit.ly that allows content providers to offer shortened links that resemble the actual domain name more closely (i.e., TechCrunch.com becomes tcrn.ch), and how this makes it appear on the surface that the proportion of tweets using bit.ly has decreased quite a bit.

bit.ly is still the most-used link shortener on Twitter, according to this lovely pie-chart created by tweetmeme.com and refreshed daily. But bit.ly accounts for 50% or so of shortened links on Twitter, and the next most popular service, tinyurl.com, only 5%. So, back to thinking about my data collection script, I have a couple of options for resolving the URLs I am parsing out of tweets. I can stick with the most popular link shorteners as reported by tweetmeme.com, use the available APIs provided by those services to look up the original URLs, and end up throwing out 30-50% of links. Or, I can resolve all of the links I find, to figure out if they redirect to a different place or not. I’m leaning towards resolving all the links at this point, but will have to do some testing to make sure it will actually work the way I think it will work.

The first part of my rant is about how the people providing these datasets don’t give me enough information to decide whether or not the dataset will be useful to me before shelling out hundreds of dollars or filling out a lengthy form or pestering my supervisor for approval.

For example, take a look at the “Twitter Census: Hashtags, URLs, Smileys by Hour” dataset on Infochimps. They want $300 for this dataset without providing a clear description of what it contains, or even how big it is. Nor is there much documentation about how these data were collected, and what, if any, sampling bias may exist.

And the free datasets aren’t really usable either—the “Twitter Census: Tweets by Hour Tweeted” dataset has the same problem as the “Hashtags…” one; in addition, when you download the file, you get a tab-delimited text file that contains four columns with no labels! Come on, people, what am I supposed to do with this? Guess?

Regarding the Yahoo! Labs data, one must fill out a lengthy form, including providing a phone number, and describe the “proposed research project”, before even being told what types of data might be available. To actually obtain access to a dataset, one must enter the name and email address of their department head, presumably to provide some kind of accountability or oversight of how the data will be used. I definitely don’t want to bug my boss before I have even seen a snippet of the data.

Places like ICPSR at the University of Michigan have been making data available for a long time; there should be plenty of examples for organizations like these to follow regarding what kind of information researchers need to determine whether a dataset will be useful.

I also want to briefly consider the difference between “raw data”, “dataset”, and “sample”. I don’t know how others think about using these terms to represent chunks of data they happen to be working with. Take this description of the Yahoo! Labs delicious.com dataset:

This dataset represents 100,000 URLs that were bookmarked on Delicious by users of the service. Each URL has been saved at least 100 times. For each URL, the date that it was first bookmarked by a Delicious user is indicated, along with the total number of saves. Also indicated are the ten most commonly used tags for each URL, along with the number of times each tag was used. This dataset provides a view into the nature of popular content in the Delicious social bookmarking system, including how users apply tags to individual items.

To me, this describes a very specific, targeted sample taken from the entire delicious.com database (i.e., the “raw data”). I think of a “dataset” as something smaller than the whole database, but without the degree of aggregation seen in the “samples” I mentioned above (i.e., counts of tweets per hour rather than the actual tweets, and tag usage counts rather than the actual bookmarking history). Maybe I am the only one thinking about it in this way.

Ultimately, the Yahoo! Labs delicious.com sample is not very useful to me. For our CSCW 2008 tagging paper, we also limited our sample of URLs to those that were bookmarked by 100+ people. But, we scraped delicious.com to obtain the *complete* bookmark histories for each URL, including the timestamp and ALL tags applied by every single user that had bookmarked the web page. The temporal information and complete history were very important for our analysis. However, since we were scraping, we only obtained the data for 30 URLs; I’d love to be able to do this kind of analysis with a bigger sample, so that we might be able to find clusters of people who all bookmarked the same URLs. But alas, this dataset from Yahoo! Labs is not suitable for this purpose.