In the last year of so of grad school, several of my fellow students at a similar stage in the program started doing “paper swaps” before a big deadline. This was an awesome idea brought to us by @jennthom. Each person who was submitting a paper agreed to review at least one other paper, in exchange for feedback on their own paper. This brilliant plan had many benefits: it encouraged each of us to finish things a *little* bit earlier than we would have otherwise, we got to learn more about what our colleagues were working on, and of course we both received feedback on our own papers and got to practice giving feedback to others. The main drawback was that it created more work at an already busy time.

An added benefit not obvious at first was that when it came time to write rebuttals to reviews for submitted papers, we had a group of people who were familiar enough with the papers in question that we could read each others’ reviews and make suggestions for the rebuttals. The great thing about this group of people was that it seemed like nobody was overly sensitive about sharing their reviews — and I think that this was a great learning too for all of us.

I have two questions based on this reflection about paper swaps and sharing reviews, and I’d love feedback if anybody happens to notice this post and wants to share:

1. How do I get something like this started at a new institution? I think what we did in grad school worked because we were a fairly small group who both trusted each other to be helpful, and were in serious need of feedback. I certainly learned a LOT from the experience, and think it would be super valuable for other students to participate in something similar. But how do I convince people the extra work is worth it, and that there is nothing to fear from sharing reviews? To that end, I am perfectly willing to share my own reviews on both accepted and rejected papers, which brings me to my next question…

2. Is it appropriate to share publicly, like on the Internets, reviews for one’s own papers? Would it just be too confusing for people if there were multiple versions of a paper, or even papers that never ended up being published, available on an author’s website along with the accepted papers (even if there were a separate page for them or something)? Would anyone even be interested in seeing these things? Also, do reviewers expect that what they write will be held in confidence? Personally, I always write reviews (and everything else for that matter) as if I am writing for an unknown, public audience — it is so easy to share these things, you never know who might see them. And I don’t want to say anything in a review that I would be unwilling to say to someone in person. I just have no idea how others feel about this.

I definitely don’t mean to imply that first year students can’t craft excellent elevator speeches. Of course they can! But it is definitely something that’s a lot easier to do now that I have plenty of previous work and research plans I am really excited to talk about.

Since I can’t critique my own first attempt at crafting an elevator speech, instead I offer these links I found on the Internets providing advice on how not to sound as lame as I did initially:

http://delicious.com/bierdoctor/elevatorspeech

For the experiment we are using the same web application created for my dissertation research, with a few small tweaks, and a new set of materials. Whenever you’re doing a study that involves participants using a prototype or other system built specifically for the experiment, it is imperative to do a lot of testing. The last thing you want is for the results of the study to reflect bugs or usability problems and not the actual phenomena of interest. So, before using the experiment app for my dissertation research, I set aside plenty of time for testing and recruited people to bang on the system and try to break it.

This time around, the tweaks to the system were so minor that I basically tested use cases that involved the new features, and nothing else. I figured not much had changed, so I could assume what worked before would still be working. This, as it turns out, is an assumption that doesn’t hold true in the wonderful world of web application development. With a web application, it isn’t just the application code itself you have to worry about. About a year has gone by since my initial data collection, and in that time web browsers have gone through several rounds of updates and major releases. Also, we’re using a different web server this time around. And finally, there’s been an update to one of the toolkits the application uses for the file-and-folder interface. So in reality, a LOT has changed from a year ago.

Fortunately, in the first experiment session we uncovered a minor “race condition” bug that hadn’t presented itself in either my dissertation data collection, or testing for this experiment (I say “fortunately” because we discovered the problem early). A race condition exists when multiple related (but separate) requests are sent from the client to the web server. Because these are *separate* requests, there’s no explicit sequencing, and unpredictable or undesirable application behavior can result if/when these requests are processed in the wrong order. This was a simple bug to fix, and so far no other bugs have presented themselves.

The reason I am glad I’m not a web application developer, is with all these infrastructural components that can change (browsers, servers, toolkits…), keeping a web application working seems to be like hitting a moving target. Firefox 3.6 included optimizations to speed up javascript, for example, which may have contributed to the race condition bug in the experiment app. A new version of Internet Explorer was released, and the toolkit the experiment app uses also released a new version with changes based on the changes to IE. It amazes me that Gmail and all those other web apps I use on a daily basis continue to work at all!

So my advice to anyone considering using a home-grown web application in their research is, come up with a test suite, document it, and run through all the test cases *every time* you intend to use the application in a new study. Even if the application itself hasn’t changed.

“People thought what they had was an address book for an e-mail program, and Google decided to turn that into a friends list for a new social network,” said Marc Rotenberg, executive director of the Electronic Privacy Information Center, an advocacy group in Washington. “E-mail is one of the few things that people understand to be private.”

Mr. Rotenberg said that his organization planned to file a complaint with the Federal Trade Commission claiming that the Google’s use of e-mail conversations to build a social network was unfair and deceptive.

1. Use information to provide our users with valuable products and services.

2. Develop products that reflect strong privacy standards and practices.

3. Make the collection of personal information transparent.

4. Give users meaningful choices to protect their privacy.

5. Be a responsible steward of the information we hold.

Sheldon & Lyubomirsky (2006). How to increase and sustain positive emotion: The effects of expressing gratitude and visualizing best possible selves. The Journal of Positive Psychology, 1(2): 73–82

Murray, Holmes & Griffin (1996). The Self-Fulfilling Nature of Positive Illusions in Romantic Relationships: Love Is Not Blind, but Prescient. Journal of Personality and Social Psychology, Vol. 7 I, No. 6, 1155-1180

Lyubomirsky, King & Diener (2005). The Benefits of Frequent Positive Affect: Does Happiness Lead to Success? Psychological Bulletin, Vol. 131, No. 6, 803– 855

now, take a look at this video in which advice in the same vein as those research findings is dished out on CNN (by someone who is apparently neither a research psychologist or therapist):

the first time i watched it, i got a pretty good laugh out of it. the perky pessimism with which Chatzky says “just fake it” is pretty awesome, and the exchange at the end with the CNN guy is hilarious. the best part, though, is where Chatzky says, “your boss thinks that you’re interviewing, because why else would you be smiling?” uhhh, ok….

but the underlying message, which i think is supposed to be self-empowering, actually ends up being pretty depressing. it’s more like, “so you’re stuck in your miserable job; you have no choice but to stay so you can keep collecting a paycheck. but if you pretend to like it maybe it won’t suck as much!” sounds hard to believe, doesn’t it?

i was particularly interested in ordering a couple books on academic writing. most of these “how to be a better writer” type of books end up telling me stuff i already know, but sometimes that’s exactly what i need. anyway, one of the books i ordered came in the mail today, called “How to Write a Lot”, by Paul Silvia.

this book rocks.

not because it contains earth-shatteringly brilliant advice, but because it is so matter-of-fact about it. for example,

Academic writing should be more routine, boring, and mundane than it is. To foster a mundane view of writing, this book says nothing about the “soul of writing”, the nondenominational “spirit of writing”, or even the secular “essence of writing”. Only poets talk about the soul of writing. You should write like a normal person, not like a poet and certainly not like a psychologist. And this book says nothing about anyone’s insecure feelings of “defensiveness” and “avoidance”; go to your local bookstore’s self-help section for that. How to Write a Lot views writing as a set of concrete behaviors, such as (a) sitting on a chair, bench, stool, ottoman, toilet, or patch of grass and (b) slapping your fingers against the keyboard to generate paragraphs. You can foster these behaviors by using simple strategies. Let everyone else procrastinate, daydream, complain—spend your time sitting down and moving your mittens.

awesome.

normally i would not buy a book with the word “self-help” in the title. i’m not sure the genre of self-help books has actually helped anybody. but i’ve gotten out of the habit of writing regularly since finishing my proposal, and i have to admit that i just feel drained and exhausted and out of things to say when i sit down in front of my laptop and try to write. so maybe self-help is really what i need!

this particular book by Boice got favorable reviews on Amazon, and i was able to find a few more reviews floating around the Internet that were also favorable. i’ll keep you posted!

- If you’re planning to have a pre-proposal meeting with your committee before the proposal defense, don’t spend a ton of time writing the proposal document in advance of that meeting. Save the effort for after you’ve received feedback from the committee.

- It isn’t necessary to analyze the heck out of a dataset right from the very beginning. Prioritize the analyses in such a way that you answer the most important or theoretically/practically relevant and interesting question(s) first. You can always go back and slice the data a different way later.

The second bit has helped me focus my analysis of the Taboo data. I’m doing a kind of conversation analysis, and I put together a fairly long list of things I could code for.

But, after looking at some of the data with my advisor, we decided that there are really two things we want to know right away. The independent variables in this study are “shared past experience” common ground and gender; I recruited 32 same-gender pairs of participants, half of whom were friends and half were strangers.

1. Does common ground allow people to communicate more efficiently and converge more quickly on the target word?
2. Do people who have common ground take advantage of it when forming their referring expressions?

The objective is to compare ‘referring expressions’ under different common ground conditions. Each clue and guess can be considered a referring expression. I’m also interested in utterances that play a feedback role, because some people might be better at providing feedback than others (and feedback is an important part of social language use).

So with this in mind, my first pass through the data will be coding utterances into clue, guess, or feedback. I will also identify instances where common ground is used. Each word for each pair will be coded as success, fail, pass (if the participants passed on the word), or error (if the clue-giver used one of the illegal words). All of this should only take me a couple of days, since the data is already cleaned! Hooray!

- A dissertation that consists of multiple studies should have one study that is the ‘main’ part of the dissertation, that makes up the bulk of the work. It is ok to have pilot data or preliminary data collection, but it is important that the dissertation not end up looking like the accumulation of a couple smaller, conference-paper-type studies.

- The Taboo study is interesting and definitely relevant, but it might be better not to include it as part of the dissertation — it might confuse people because it was collected at a different time and for a different purpose.

- I have set a rather aggressive timetable for myself; I should think about defending around February 2009 rather than doing it in the Fall term of 2008. That way I can concentrate on getting a couple of papers from my dissertation work out the door, which will (hopefully) make me a stronger job candidate.

- Research topics tend to come and go in cycles, and at this point there don’t seem to be a lot of people in HCI or LIS looking at the same thing I am — but my topic area seems to be gaining momentum again, so I have a good opportunity to make an impression with my work.

I think it will be interesting to look back on these notes at the end of the process — I am guessing that they’ll be more meaningful in hindsight than they are right now.