How does social language use affect information sharing in user-contributed content repositories?

I’ve been going back and forth on whether I want to say information seeking or information sharing, and I think I’ve finally settled on sharing. I feel like seeking implies a focus on things like information needs and relevance judgments, and I am more interested in focusing on the implications of language choices for the sharing of information among group members, or users of a particular system.

Sharing can be thought of in two different ways. The first is “to tell someone”, as in sharing a secret. The second is “to have in common”, as in a shared experience. Email is one technology for sharing information that operates on the “to tell someone” principle. However, sharing may also be accomplished in the “have in common” sense via online repositories, where group members are able to find and access common information when it is needed.

I also have three specifying questions, corresponding with (I hope) parts of my dissertation:

1. How does common ground (from shared past experience) affect language choices in a collaborative naming task?

2. How do groups organize and share information via an online repository of user-contributed content? What problems do individual members encounter during information seeking within the repository?

3. How do the source of common ground and the intended audience affect filename choices, and the effectiveness of those filenames for supporting information sharing?

The first research question asks how common ground, a factor that has been shown to affect social language use in many different contexts, affects word choices in a synchronous collaborative naming task. The second question sets up a field study exploring how groups use online repositories, particularly for organizing and seeking information. Creating filenames for a group information repository might be thought of as a collaborative naming task, but only if group members do so with others in mind, and it is not clear whether some filenames are better than others for information sharing.

The answers to these two questions will tell me whether it is worthwhile to pursue a link between social language use and naming, by supporting or refuting the existence of such a link, and suggesting processes by which naming choices and information seeking and sharing might be related. In the third question I identify two factors that might affect naming choices and information sharing: source / type of common ground, and intended audience. The influence of these factors will be explored in a two-part lab experiment.

These questions do not address the issues of organization / structure and evolution that I have been thinking about, for practical and scoping reasons. I am working on a couple of other posts that will attempt to position these questions within a larger framework for how to approach the problem of “group information management”.

I’ve posted about why I believe focusing on language is important, but I still need to write a similar “motivating post” about my focus on information structure and organization. Slowly but surely, I’m chipping away at the ideas that will be in my proposal.

What I’m trying to do next is make the (giant?) leap from the high-level questions above, and specific questions that I’ll try to answer. I think I am getting hung up on ideas I’ve had in the back of my mind for studies that I want to do, trying to reconcile them with these research questions. What I really should be doing is brainstorming from the questions, not trying to arrive (backwards) at the questions from the study ideas. Perhaps I’ve done a little bit too much exploratory data analysis with the delicious and ctools datasets, and it’s time to get back to basics, like actually having a research question BEFORE collecting the data. What a concept.

1. How do the information structure and organization of a user-contributed content system shape its evolution and use?

Two characteristics of user-contributed content systems are particularly relevant to this question. The first is that the information/content within the system is the point of using the system. Contributing and viewing or accessing content are the primary activities that take place in the system. The second is the content and structure present in the system at a given point in time constrains future use, in terms of what is available for them to access, what others choose to contribute, where they put the new content, and how they label or tag it. Small, individual choices accumulate into larger patterns that affect the choices and behavior of future users.

I want to know if there are patterns to be found in the process, for example, what might affect the growth of the system; what are people thinking when they make labeling or tagging choices at various stages in the development of the system; tracing the consequences and constraining effects of different types of changes on future usage; what information or cues do people attend to when trying to figure out where to” put” something, and how to find it (or re-find it); what aspects or “features” of the content make it into the labels or tags; how the “mental model” of one’s group members changes and develops over time; how and why conventions or explicit patterns form (is it really a kind of collective linguistic problem-solving?). Lots of questions, and the opportunity to compare-and-contrast among different groups in the field, using a particular system, would allow me to look for patterns at three different levels of analysis: individual, within-group, and between groups. Yay, data!

2. How do users establish common ground and negotiate meaning when contributing and seeking information asynchronously?

This question comes from the psychologist in me. I have a hypothesis that the processes at work when people use language to communicate might also affect language use in other situations that are not as obviously “communication”. Brennan (1998) suggested that we might react to computers as though they were human communication partners; she gave the example of a user of a command-line system who uses the command “ls” often to check on the status of the system, because system status is not provided automatically. I want to find out whether people’s language choices in a user-contributed content situation follow patterns that can be predicted from what we already know about language use in communication. Either way, the results will inform improved design choices, and also contribute to mediated communication theory. I intend to answer this question through lab experiments designed to identify various factors that contribute to tag, label, and description choices in user-contributed content systems.

3. How do information organization and language use affect information seeking and access in user-contributed content systems?

This question is about investigating outcomes for information consumers, in a more systematic way, once I’ve invented a design or other intervention and am ready to test it. Given my timeframe, this goes under the “future work” category.

1. How do information structure and language use shape the evolution of a user-contributed content system? The key to this question is “evolution”. It is important to collect field data over a period of time to begin to understand how language is used to structure and seek information, and to communicate in user-contributed content systems. I’m not sure yet exactly what data I might collect — but I do know that retrospective interviews will not be enough. From my CTools pilot I’m learning that language use in user-contributed content systems seems to be tacit in the same way language use in conversation is. One can make up a reason for saying something a certain way after-the-fact, but the processes taking place in the moment are unconscious. The challenge will be to collect data that is detailed enough to piece together what is going on. My initial feeling is that this will consist of a combination of interviews, observations, and think aloud protocols. How should I focus the data collection?

2. How do people establish common ground and negotiate meaning when contributing and seeking information asynchronously, in a shared environment? This question asks specifically about language use and adaptation. It is impossible for people to create isolated labels for things that are completely unambiguous, because people use contextual cues and rely on interactions with each other to understand what words mean (Lansdale, 1988). How do people cope with language ambiguity in these systems, and understand one another? In conversation, people are able to provide each other with feedback (in synchronous communication), or carefully plan their utterances (in written discourse or other texts). User-contributed content systems do not afford either of these processes.

The image below is a rough draft of another, more specific version of my concept diagram. It specifically refers to factors that might affect word choice, and from which hypotheses could be generated and tested experimentally. The challenge here lies in deciding where to start. My intuition is to focus on precedent, awareness, and model of others’ knowledge. It is important to remember both participants in the communication: the producer and the consumer.

3. What are the consequences for information seeking and access? This is something I don’t think I’ll get to in my dissertation, because I would have to build and test a system. This is my eventual goal, but does not fall within my time-frame for graduating.

Unlike people engaged in monologue or in reading or writing text, conversationalists have the opportunity to rely on their partners in ways that structure the discourse itself (p123).

This sentence captures the most obvious difference between spoken and written discourse. People who are speaking with one another in person or via some communications medium, like the telephone or instant messaging, are able to iteratively negotiate meaning and repair miscommunication. In less synchronous interactions, conversation partners must act more independently. Leaving a voicemail, sending an email, and writing in a blog are examples of discourse that are structured and produced without much opportunity for interaction and feedback. And yet, people are still able to understand one another, by making inferences about meaning based on their knowledge about the situation, their conversation partner, and past interactions.

In social software systems, users both contribute and consume content. The textual labels, tags, and descriptions associated with the content make up the structure of the information in the system as it is seen by the user. This “text” is an artifact that evolves over time as information producers and consumers iteratively and asynchronously contribute, organize and seek information. How does the evolution of form and content take place? How do people establish common ground and negotiate meaning in these systems, when creating labels, descriptions, and tags? And, what are the consequences for information seeking and access?

Understanding how groups collaborate to organize, label and share information online, in order to design tools that will make finding and accessing user-contributed content easier.

Sounds pretty good, on the surface, but it is rather broad, and doesn’t address the methods I’ll be using to approach the problem. I once suggested to somebody else that there are three things s/he needed to figure out before proceeding:

1. The problem I want to solve (or understand better), and why
2. The framework I’m using to think about the problem
3. My research question(s) and the data I’ll need to answer it(them)

Well, I have ideas about all three, but NOW is the time to get concrete.

The Problem, in a nutshell, is this: there’s all this user-generated content out there, creating new opportunities and difficulties for people seeking information. I’m not going to call it a revolution, it’s just that the tools and the infrastructure both exist that enable information producers to generate information and make it available to anybody with a network connection. The barriers to entry are a lot lower, and so there’s a lot more information out there (this blog included — I mean, I installed this WordPress server by clicking a button). Tools exist in the world already that connect information consumers with the information they need, like library catalogs, yellow pages, human intermediaries, etc. However, these tools all imply a central authority of some kind. The amount of centralized control in user-contributed content systems varies, but generally isn’t a lot. And as I’ve said before, the dynamic and collaborative nature of the content available online today is a new twist on how people have used technology to share information in the past. The technology is new, and the tools are evolving — people are just starting to figure out how it works and how they can make it work for them.

The high-level Framework: the thing about user-generated content, is it seems to be an inherently social endeavor (like a lot of things people do with the Internet). People are sharing information like crazy, through many different kinds of online tools set up for this sort of thing: blogs (360.yahoo.com), photos (flickr.com) and videos (video.yahoo.com), questions and answers (answers.yahoo.com), social bookmarking (del.icio.us). And those are just the sites owned by Yahoo! People call it “social computing”, but what does that really mean? One way to look at it is that it all boils down to communication through language. Some people are creating, structuring, and organizing the information for others to access. I believe that this exchange can be though of as a kind of communication, and it can be analyzed as a form of discourse. There are, of course, some interesting similarities and differences to more synchronous, iterative forms of communication, which I will go into in a different post. But in short, I believe a “social language use” perspective on the structuring and organizing of user-contributed content can help us to better understand people’s behavior today, and design the next generation of tools.

Some Research Questions: I am the least sure about this part at the moment, mostly because I have so many questions and potential studies circulating around in my head, and I’m still working on scoping a reasonable dissertation. Again, at a high level, there are a few things I want to figure out. How do social language processes shape the structure and organization of information in social computing systems, and how people are able to find and access the information they seek? I will use both lab studies and fieldwork in answering these questions. Now I just need to break it down a bit further and be more specific.

we calculated interuser agreement for a sample of 349 websites from del.icio.us. in order to do this, we first created a user x user matrix for each web page consisting of all the users who had ever bookmarked that web page. for some of the more popular web pages, such as slashdot which had been posted by 22,193 users at the time the poster was presented, it was necessary for the sake of computation time to use a random sample of users who had bookmarked the page rather than the entire list of users. each cell in the user x user matrix contained a percentage representing what how many times those two users applied the exact same tags for that web page, out of sum of unique tags used by that pair for that web page. so if user1 applies tags a, b, c, d and user2 applies tags a, c, e, f, g, the interuser agreement is 2/7.

we then averaged all the cells in each matrix to produce the average interuser agreement for that web page. the frequency histogram below shows a summary of the results we obtained for our sample of 349 web pages:

on average, the interuser agreement for the sample of web pages we obtained from del.icio.us was 17%. it doesn’t seem like an earth-shattering finding, but as i said it generated a lot of conversation, and a couple of good ideas for follow-up research:

1. has interuser agreement increased over time on del.icio.us?
2. are there groups of users who have higher than average interuser agreement?
3. what does interuser agreement look like on other sites that use tagging, like flickr?

we have the data that we need to start working on the first question, since we parsed the entire posting history for each web page including date, user, and tags applied each time the page in question was bookmarked by someone on del.icio.us. the argument made by people at the conference was that the “tag suggestion” interface has changed over time, so a more recent time interval may show higher interuser agreement due to a greater degree of tag-reuse.

the second one is a little bit harder, since we don’t have access to the entire del.icio.us database. one hypothesis is that people who have more common ground (same friends, similar interests, etc.) might be more likely to use the same language to refer to things, and therefore might have higher interuser agreement. the difficulty here is in finding a proxy for common ground that can be quantified, and is represented in the data stored by del.icio.us. one idea is the del.icio.us subscription feature — users are able to ‘subscribe’ to other people or specific tags (or combinations of both). it would be possible to find pairs of users who subscribe to each other or to the same tag, and then calculate their interuser agreement for pages they both bookmarked. however, without access to the entire database, our data collection consists of what we can brute force scrape from del.icio.us more or less randomly. how do we identify people who fit the subscription criteria, and who have also bookmarked the same web pages? we would need a pretty darn big sample, and the entire posting history of each user (which is difficult to scrape).

data collection issues aside, the question of what might serve as a useful proxy for common ground is important. finding a measure that actually reflects what we think it does is essential — otherwise, there will be too much noise due to “measurement error” in the data for us to detect any meaningful results. let’s say two people both subscribe to the same tag, but have no bookmarks in common. do they share the same “level” of common ground as another pair who subscribe to the same tag but have 10 bookmarks in common? what about 50 bookmarks in common? what if we don’t even need subscription data, and common bookmarks is a suitable proxy? there’s no way to know without collecting more data: a sample of del.icio.us users, their complete bookmark histories and current subscriptions/network, and complete posting histories for all of the pages posted by those users. and in order to collect data, we must figure out how to sample without the equivalent of a comprehensive del.icio.us user directory.

the third question, above, is harder still. flickr is a site that allows people to tag photos, but it is unclear whether people are tagging the photos uploaded by others, or just their own. interuser agreement on tags for a given photo might not make sense in terms of the way people use tags on flickr. it might be more important to look at agreement in reverse — how many different kinds of images are returned when i search for ‘snowball’ on flickr? just now, the first 24 images returned by that search consisted of photos of pets, flowers, a snow cone, balls made out of snow, snowball fights, and amigurumi knitted stuffed animals.

so, is interuser agreement even a useful concept for flickr? the argument for looking at interuser agreement in del.icio.us is that people use del.icio.us to find, store, and organize information produced by others, and the language (tags) used to represent that information are the means by which the information is accessed. my guess is that flickr is primarily used for sharing and browsing one’s own photos and the photos of friends and family in which case ‘contacts’ would be more important than ‘tags’. flickr also has the concept of ‘groups’ or ‘pools’ which employ intermediaries to moderate the content — something del.icio.us does not do for tags. if we embark on collecting data from flickr.com, it will have to be used in conjunction with interview data about how people use flickr or it will be difficult to interpret the quantitative results.