The dataset consists of conversations between dyads that took place while they completed two different interactive tasks. The conversations were recorded, transcribed, and segmented into utterances according to some criteria. This means that there are repeated utterances from each participant, and from each dyad. Different research areas use different terms to refer to this kind of setup: repeated measures, panel data, clustered data, etc. The analysis is further complicated by the fact that the predictors and variables are all categorical. Some are binary, the presence or absence of something. The more interesting variables have more than two categories (in some cases, MANY more).

I am trying to estimate the strength with which each of a set of 15+ utterance goals is associated with one of three roles participants assumed as part of the study. To do this, I need to specify a mixed-effects multinomial logit model, with a set of fixed-effects categorical predictors and a hierarchical random effects control for participant within dyad. This involves choosing a reference category of the response variable, and then running a series of binomial logit models that compare all the other levels of the response variable in turn with the reference category.

Here is where I am running into a situation, again, where I am pushing up against what mainstream statistical software packages are reliably capable of, and even R does not seem to be able to do what I want without more programming than my meager statistical background has prepared me for. The problem as I understand it is, each one of the binomial logit models that makes up the multinomial results uses a different subset of the data, excluding those observations that are related to the levels of the response variable not included in the model. This means that the random effects are estimated differently for each binomial logit model, depending on which observations are included in the subset. The upshot of all of this is the overall multinomial model estimates come out differently, depending substantially on which category is chosen as the reference category.

So that’s the problem. However, I did not write this to whine about how I am stuck. I’ve been trying to figure out a solution that I can live with… do I bail completely? Hire a real statistician? How can I figure out how biased the results would be if I were to to do a purely fixed-effects model? (Without random effects controls, any results produced might in fact be due to some unique aspect of the conversation within a particular dyad in a particular role, rather than indicative of something that shows up across all of the dyads.)

Researchers in many fields work with categorical data, and at least some of them over the years must have encountered this problem, whether they knew it or not, and were faced with the same tradeoffs. In order to get the paper out the door they had to just pick a compromise and go with it. But, any results reached due to a compromise are biased in some way. Models like this are just now becoming possible for people like me, with just enough stats knowledge to be dangerous, to run using fairly standard statistical software packages. But what about all the research that has come before — how accurate are those models, and the results they produced? How much do people allow what is statistically feasible to determine their research design, vs. compromising on the analysis after the fact? We all stand on the shoulders of giants, but how often were the giants using naive or incorrect statistics?

I recently came across mention of an alleged “censorship” controversy related to Digg.com [see here, and here for mentions in mainstream media], in which a group of coordinated users apparently succeeded in preventing certain stories they found politically objectionable from reaching the front page of Digg, so these stories would not receive wide exposure. The users achieved this end through what is essentially a thumbs up / thumbs down mechanism fundamental to the way Digg works, by which users vote on whether stories should be promoted or buried. As one blogger points out, opinions about censorship aside, these users were operating within Digg’s available functionality and did not necessarily violate any rules. People who are upset about this use of Digg’s voting mechanism claim the group of users were gaming the system — coordinating which stories to target via another social media application (Yahoo! Groups).

The “gaming the system” and “censorship” aspects of this controversy are less interesting to me personally, than the flexibility of such a simple voting mechanism, used to express an entire political agenda rather than individual, personal preferences. This is an instance of the point I was trying to make (badly?) a few days ago — that even simple mechanisms can be tools for expressing a wide variety of meaning, but that meaning is not obvious from single contributions. In this case, the coordinated intentions of the group only became apparent in aggregate, and only after people who were pissed off about having their stories consistently targeted for “burial” were motivated enough to figure out what was going on. In other words, the meaning behind these actions was not present in the aggregate voting data; was only visible if you already knew where to look.

A URL shortener is an online service that assigns a short URL to a web page’s original address, such that the web page can be accessed either by the original URL or by the new, shorter URL. People use these services in conjunction with Twitter, when they’re including a URL in a tweet. Because tweets can only be 140 characters long, a shorter URL means more characters left over for saying other stuff.

When I say there is a plethora of URL shorteners available, I truly do mean it in the “extreme excess” sense of the word. For example, there’s a Flickr set of screen captures that contains images of the home pages of 129 different link shorteners, including shadyurl.com for when you want your “shortened” link to be suspicious and frightening. Bizarre.

Speaking of shady URLs, one issue I have with URL shorteners is that I am never sure what I am going to get when I click on a link from a service like bit.ly, which is the “default” URL shortener of Twitter. Phishing attacks in social network applications (like Facebook and Twitter) are becoming more common, and tricking people into clicking on links that execute code intended to steal passwords, etc., is often the goal.  As a result, I rarely ever follow links that show up in my Twitter feed, or in my email. It just seems to risky to me to click on links from URL shorteners that disguise the ultimate destination.

Interestingly, the distribution of Twitter posts using different URL shorteners has shifted quite a bit over time, as described by a blogger on TechCrunch.com in “What happened to bit.ly’s market share?” The article describes a new “pro” service offered by bit.ly that allows content providers to offer shortened links that resemble the actual domain name more closely (i.e., TechCrunch.com becomes tcrn.ch), and how this makes it appear on the surface that the proportion of tweets using bit.ly has decreased quite a bit.

bit.ly is still the most-used link shortener on Twitter, according to this lovely pie-chart created by tweetmeme.com and refreshed daily. But bit.ly accounts for 50% or so of shortened links on Twitter, and the next most popular service, tinyurl.com, only 5%. So, back to thinking about my data collection script, I have a couple of options for resolving the URLs I am parsing out of tweets. I can stick with the most popular link shorteners as reported by tweetmeme.com, use the available APIs provided by those services to look up the original URLs, and end up throwing out 30-50% of links. Or, I can resolve all of the links I find, to figure out if they redirect to a different place or not. I’m leaning towards resolving all the links at this point, but will have to do some testing to make sure it will actually work the way I think it will work.

The first part of my rant is about how the people providing these datasets don’t give me enough information to decide whether or not the dataset will be useful to me before shelling out hundreds of dollars or filling out a lengthy form or pestering my supervisor for approval.

For example, take a look at the “Twitter Census: Hashtags, URLs, Smileys by Hour” dataset on Infochimps. They want $300 for this dataset without providing a clear description of what it contains, or even how big it is. Nor is there much documentation about how these data were collected, and what, if any, sampling bias may exist.

And the free datasets aren’t really usable either—the “Twitter Census: Tweets by Hour Tweeted” dataset has the same problem as the “Hashtags…” one; in addition, when you download the file, you get a tab-delimited text file that contains four columns with no labels! Come on, people, what am I supposed to do with this? Guess?

Regarding the Yahoo! Labs data, one must fill out a lengthy form, including providing a phone number, and describe the “proposed research project”, before even being told what types of data might be available. To actually obtain access to a dataset, one must enter the name and email address of their department head, presumably to provide some kind of accountability or oversight of how the data will be used. I definitely don’t want to bug my boss before I have even seen a snippet of the data.

Places like ICPSR at the University of Michigan have been making data available for a long time; there should be plenty of examples for organizations like these to follow regarding what kind of information researchers need to determine whether a dataset will be useful.

I also want to briefly consider the difference between “raw data”, “dataset”, and “sample”. I don’t know how others think about using these terms to represent chunks of data they happen to be working with. Take this description of the Yahoo! Labs delicious.com dataset:

This dataset represents 100,000 URLs that were bookmarked on Delicious by users of the service. Each URL has been saved at least 100 times. For each URL, the date that it was first bookmarked by a Delicious user is indicated, along with the total number of saves. Also indicated are the ten most commonly used tags for each URL, along with the number of times each tag was used. This dataset provides a view into the nature of popular content in the Delicious social bookmarking system, including how users apply tags to individual items.

To me, this describes a very specific, targeted sample taken from the entire delicious.com database (i.e., the “raw data”). I think of a “dataset” as something smaller than the whole database, but without the degree of aggregation seen in the “samples” I mentioned above (i.e., counts of tweets per hour rather than the actual tweets, and tag usage counts rather than the actual bookmarking history). Maybe I am the only one thinking about it in this way.

Ultimately, the Yahoo! Labs delicious.com sample is not very useful to me. For our CSCW 2008 tagging paper, we also limited our sample of URLs to those that were bookmarked by 100+ people. But, we scraped delicious.com to obtain the *complete* bookmark histories for each URL, including the timestamp and ALL tags applied by every single user that had bookmarked the web page. The temporal information and complete history were very important for our analysis. However, since we were scraping, we only obtained the data for 30 URLs; I’d love to be able to do this kind of analysis with a bigger sample, so that we might be able to find clusters of people who all bookmarked the same URLs. But alas, this dataset from Yahoo! Labs is not suitable for this purpose.

Regarding the prevalence of online news consumption, social networking site user, and status update use:

• 61% of Americans get news online in a typical day… just 2% rely exclusively on the internet for their daily news.
• 43% of Americans use social networking sites such as Facebook, MySpace or LinkedIn – and 97% of them are online news consumers.
• 14% of Americans use Twitter or other status update functions. Virtually all (99%) are online news consumers.

Regarding where internet users (75% of survey respondents) get their “socially filtered” news online:

• 71% get news forwarded to them through email or posts on social networking sites
• 28% get news from people they follow on social networking sites like Facebook, on a typical day
• 6% of all internet users get news via Twitter feeds

Regarding where internet users share news online:

• 48% pass along email links to news stories or videos
• 17% have posted links and thoughts about news on a social networking site like Facebook. That translates into 30% of social network site users.
• 3% have used Twitter to post or re-Tweet a link to a news story or blog. That amounts to 18% of Twitter users.

And finally, a breakdown of where “online news users” (71% of survey respondents) get their news:

• 75% get news forwarded to them through email
• 30% get news from someone they follow on a social networking site like Facebook
• 6% get news from someone they follow on Twitter
• 7% get news from a news website such as Digg where users rank stories
• 11% get news from the website of an individual blogger who does not work for a major news organization

One difficulty I had when reading this report was keeping the segmentation of the sample straight with respect to the percentages reported. For example, in the bullet points above, I mentioned “Americans”, “internet users”, and “online news users”, and sometimes it wasn’t clear to me which sub-sample a particular percentage referred to. So if you see something that seems like it doesn’t quite add up, that’s probably why.

Another confusing thing is that Twitter is in its own category for most of the results in the report, separate from social networking sites. This is interesting to me, since Facebook is lumped together with other social networking sites (like MySpace and LinkedIn). But there was at least one question about “status update functions”. I haven’t looked up how people were asked about this, so I don’t know whether it lumps Facebook and Twitter together.

What stood out most to me in the report was the contrast between my perception of the Twitter hype in the news media (i.e. OMG! Twitter! Everybody’s doing it!), vs. what percentage of Americans are actually using Twitter. I also found it interesting that there’s such a large difference between the proportion of internet users who get news via Facebook vs. Twitter (28% vs. 6% of internet users). The report includes some information about what these so-called “news participators” (37% of internet users) are like:

News participators are information omnivores and technophiles. They stand out from the pack in the same way as those who have set up their cell phones to be “on alert.” In fact, among news participators, 19% have news alerts sent to their cell phones. News participators are fond of social media: 76% of news participators use social networking sites; 34% of news participators use Twitter, and 26% of news participators are bloggers. The average participator uses 4-6 media platforms on a typical day; seeks out nine or more news topics online; and surfs 3-5 different kinds of news websites on a typical day.

The typical online news participator is white, 36 years-old, politically moderate and Independent, employed full-time with a college degree and an annual income of $50,000 or more. Interestingly, while white adults make up the bulk of the online news participator population, black internet users are significantly more likely to be news participators than their white and Hispanic counterparts. Almost half of black internet users (47%) are news participators, compared with just 36% of white internet users and 33% of Hispanic internet users. Not surprisingly, the youngest internet users (18-29 year-olds) are more likely than their older counterparts to be online news participators, with just under half of that age group (46%) contributing to the creation, commentary, or dissemination of news online. Men and women are equally likely to participate in online news production.

The research question/motivation for the analysis stated in the paper is, “What is the role of randomness in human behavior and to what degree are individual human actions predictable?” The main finding is that people’s daily mobility patterns are very predictable, even when they travel quite far on a daily basis.

I suppose researchers have not been able to model human mobility quite like this before—that’s a LOT of data, even the sample of 50,000 they used for the analysis reported in this paper. But I’m not sure I understand the claim that this finding is surprising: “Yet it is not the 93% predictability that we find the most surprising. Rather, it is the lack of variability in predictability across the population.” How surprising is it that humans are creatures of routine and habit? What was more surprising to me about this paper was the idea that “current models of human activity are fundamentally stochastic”, i.e. the previous status quo was the assumption that there is an important random component to human activity.

The paper seems to be saying “look, the assumptions made by people who study this kind of thing are WRONG, and we can prove it”. I typically like those kind of papers. But my feeling about this one is, while it is useful to know that people’s mobility patterns aren’t random if you study these kinds of things, it also seems like a giant WELL, DUH. I suspect this paper is receiving attention because of the sexy dataset, and not because it presents counterintuitive results. (I first heard about the paper on Twitter.)

For me, the most interesting aspect of the paper is how it glosses over what could very well be a huge sampling bias. This is not unique to this particular paper—many analyses of large “social computing” datasets have similar sampling biases due to technical constraints or dataset limitations. When one analyses an existing dataset, one must make do with the information one is given. (A related example is using IP geolocation to identify users’ locations when the only information you have about them is their IP address—there’s systematic bias in that data for sure, but in many cases there’s just no better way to do it.)

So for example, the cell phone mobility dataset contains location information only for instances when phone calls were placed (or received?)—i.e., the phone had to be in communication with a tower for the tower location to be recorded. This means the dataset contains locations for people *only when they’re using their phones*. If a person doesn’t make any calls, their location is not captured in the dataset. Are there systematic differences in mobility behavior between people who make calls and those who don’t? I’m someone who makes maybe one phone call a day, although I send a lot of text messages and use the packet data service quite a lot. Could it be possible that people who make a lot of calls have more predictable mobility patterns? While I am *sure* this possibility has occurred to the authors, the paper doesn’t address that question. I also wonder what systematic differences exist between people who have cell phones and those who don’t. But the paper doesn’t include a discussion of sampling bias, or any other potential threats to validity.

This is a big problem, I think, in the reporting of results from super large datasets. The datasets are SO big, that they are thought of as more like population data than sample data, and the results are taken for “truth” without being subjected to appropriate scrutiny. Take this blurb written about the article, from an NPR story:

A new study used cell phone billing data for 50,00 people in a European country to show that people’s travel patterns are extremely predictable. That’s true for both homebodies and jet setters. Regardless of age, language group, etc, people’s movements were predictable 93 percent of the time. The study shows the emerging power of using cell phone data for social science research. (from http://www.npr.org/templates/story/story.php?storyId=123879603)

I think it is extremely important when reporting analyses of large datasets to be exceedingly clear about issues like sampling bias and generalizability, and I’d like to see a requirement that papers address these issues. For example, this particular paper might have reported statistics on what proportion of the dataset had to be excluded due to lack of location data. Or, the authors could have undertaken a secondary data collection to try to find out whether those excluded people differed from the analyzed sample in some systematic way.

I’m not saying I think the findings of this particular paper are invalid—the results make perfect sense, and perhaps that’s why the paper doesn’t even mention threats to validity. But then, how is the finding counterintuitive if it makes so much sense we don’t even question it a little bit?

“People thought what they had was an address book for an e-mail program, and Google decided to turn that into a friends list for a new social network,” said Marc Rotenberg, executive director of the Electronic Privacy Information Center, an advocacy group in Washington. “E-mail is one of the few things that people understand to be private.”

Mr. Rotenberg said that his organization planned to file a complaint with the Federal Trade Commission claiming that the Google’s use of e-mail conversations to build a social network was unfair and deceptive.

1. Use information to provide our users with valuable products and services.

2. Develop products that reflect strong privacy standards and practices.

3. Make the collection of personal information transparent.

4. Give users meaningful choices to protect their privacy.

5. Be a responsible steward of the information we hold.

i’ve been using this service called RescueTime for the past several weeks as a way to track my hours for different projects i am working on, and as an indicator of my productivity in general. basically, you install a little app on your computer, and it sends data about what applications are active to the RescueTime server. you can log in and see reports of how much time you are spending looking at which apps and web pages (for $8/mo. you can get reports broken down by window title, not just application).

i have been happy to learn that i don’t “waste” as much time as i might have thought. but this past week isn’t a very accurate indication of my normal work habits. i went from notes and graphs to a 10-page ACM-format paper in a week:

(click for animated gif showing May 26 – June 1)

it’s nice to see that i’ve still “got it”, i guess. but that was not a fun week.

i highly recommend RescueTime, if like me you want to be more meta about how you spend your time, and like looking at data.

Each person was instructed to organize the documents with a different “target audience” in mind: for themselves, for somebody in the same graduate program, and for somebody in the other graduate program. Twenty people were randomly assigned to each “target audience” group, ~10 from each community, and each person organized the documents once, for a single target audience. This resulted in the creation of 6 different “types” of file-and-folder hierarchies, by PRODUCER and AUDIENCE; the N in the chart below represents both the number of participants and the number of hierarchies created, by type:

I have been exploring different ways to analyze the hierarchies participants created, and I am starting to think there are three types of measures:

1. vocabulary — word-level measures, like label agreement, average word rank, number of unique words, length of labels
2. “topology” — structural measures, like number and size of folders, average path length, etc.
3. semantics — this one is a little harder to measure than the others. i wanted to know whether the conceptual groupings of files might look different based on the community of the hierarchy creator, and the target audience

I used multidimensional scaling (MDS), which I wrote about a few days ago, which seemed to show that there were indeed meaningful patterns in the way documents were grouped together. But, I lost too much information with this technique — the MDS showed three distinct conceptual groups, but it was hard to determine whether structure existed within those groups.

Based on previous categorization research (Rosch et al. 1976), I expected that students from CS would create more nuanced conceptual structures for the CS-related documents, and MSI students would do the same for the information-science-related documents. but MDS was not the right technique to use for this — so I used hierarchical cluster analysis instead.

Below are two dendrograms, one that represents the clustering based on data from all of the CS students, and the other that represents data from all of the MSI students. The same three groups from the MDS are also represented here: CS, Information Science, and Security.

In the aggregate MSI student dendrogram, the Information Science cluster is broken into two parts:

In the aggregate CS student dendrogram, the same documents that make up Info Sci 1 and 2 above are merged into one cluster, while the same documents that make up the CS cluster above are broken into two groups:

My next analysis steps will be to figure out how to use this information to systematically examine all of the hierarchies for evidence of these clusters. Ideally, I would like some kind of quantitative measure that indicates to what extent individual participants created structures with these same kind of patterns — but I’m not sure how to do that yet. My ultimate goal is to be able to compare hierarchies along all three dimensions mentioned in this post: vocabulary, topology, and semantics, and find out whether differences exist according to common ground and audience design factors.