Doctoral Posters #25-28
Purpose/Objective of study: Computer support helpdesks deal with many kinds of customer requests. While many requests are routine and procedural, there may be others that are novel and lead to sensemaking situations. Besides exploring the sensemaking and handoff practices in the field, this research also tried to answer the following questions raised by theory and past research:
Sample and Setting: Ten participants from two computer support helpdesk groups at a large midwestern university were interviewed for this study. The first group (G1) worked on the phone based helpline for anyone affiliated with the university. The second group G2 provided support to engineering school members through phone in, walk in and email.
Method (Qualitative, quantitative, historical, etc): Qualitative Methods were used.
Data collection technique(s) (Interviews, questionnaires, focus groups, etc): Semi-structured interviews were conducted with the participants at their workplace. The 1-2 hour long interviews were focused on learning the work habits and work culture of the participants, the sensemaking situations they faced. The participants were also asked if there were handoffs during the course of their work, if so what were the reasons for the handoffs, what were their handoff strategies, how they decided when to handoff and systems and tools they used for handoffs
Results: Successful handoffs required deliberate effort in creating high quality handoff material, which was an expected norm at one of the groups. Also, most handoffs occurred either very early or very late in the sensemaking process. The principle of least collaborative effort (Clark & Gibbs, 1986) can be used to explain the norms of creating handoff materials and the choice of handoff time.
Purpose/Objective of study: We know much about how to help people make use of objective information, but use of subjective information has received far less attention. In part, this reflects the nature of our information sources, which in the print era were strongly filtered on cost-benefit grounds. The recent explosion of unfiltered sources such as blogs, coupled with expanded access to online news reports that characterize the attitudes of specific individuals, provides a new window into characterizing subjectivity. We are now faced with the challenge of abundance rather than scarcity – how will we help users make sense of this cacophony? Our goal is to explore the design of tools to help with that task. This involves two coupled challenges: focus and characterization. We achieve focus by using information retrieval techniques to obtain a set of documents on a topic that is of interest and then allowing the user to designate parts of those documents to serve as exemplars for aspects that they wish to see characterized. A novel feature of our work is that we allow the exemplars to be drawn from documents in two languages (English and Chinese), thus supporting exploration of multilingual collections. Segmentation and classification techniques are then used to automatically assemble a large set of document segments on those same aspects. Techniques recently developed by others (for English) are then used to characterize the “semantic orientation” (attitude, opinion, sentiment, affect, emotion) of each segment on two scales (degree of positive, degree of negative); research from psychology indicates that both can be active simultaneously. Development of comparable capabilities for Chinese is the final research goal of this project.
Sample and Setting: Intellectual property rights considerations make it advantageous to conduct initial experiments with news sources. For aspect classification, we used English and Chinese news articles from the Topic Detection and Tracking collection, which includes pre-annotated topics. For Chinese attitude classification, we obtained the newly available Chinese opinion analysis test collection by participating in the 2007 NTCIR Chinese Opinion Analysis Pilot Task organized by Japan’s National Institute of Informatics.
Method (Qualitative, quantitative, historical, etc): Model-based quantitative evaluation methods are used. For aspect classification, accuracy, precision, recall, and F-measure are reported. Precision, recall, and F-measure are also used to assess the utility of sentence polarity classifiers for Chinese.
Data collection technique(s) (Interviews, questionnaires, focus groups, etc): For aspect classification, we hired 2 information professionals to annotate document segments for aspects of selected topics in test collections from the 1999, 2000, 2002 and 2003 “topic tracking” task.
Results: For Chinese attitude classification, we have strong results for detection of opinionated sentences and sentence-level polarity classification (when compared with other NTCIR participants). Experiments on aspect classification are still underway now and expected to be completed in November.
Purpose/Objective of Study: The purpose of this study is to provide answers to a fundamental question: how useful are displayed query terms for different kinds of users? More specifically, this study addresses the research questions: 1) how does the performance of users with different levels of domain expertise and search experience differ? 2) What is the relationship between the searcher’s perceived usefulness of displayed query terms and the search outcome?
Sample and Setting: We observed four different kinds of information seekers using an experimental information retrieval system based on a 10-year subset of MEDLINE: 1) search novices; 2) domain experts; 3) search experts and 4) medical librarians. The information needs were a subset of the relatively difficult topics originally created for the TREC (TextREtrieval Conference) 2004 Genomics Track. The study was conducted in a laboratory setting within academic research environment.
Method (Qualitative, quantitative, historical, etc): The experiment was a 422 factorial design with four types of searchers, use of an experimental system with/without the aid of MeSH terms and control of search topic pairs. The order of the two versions of a system, searcher types and search topic pairs were controlled by a Graeco-Latin squared balanced design. Participants searched either using a version of the system in which MeSH terms were displayed or another version (Non-MeSH) in which they had to formulate their own terms. Based on the experimental design 32 subjects (8 for each searcher type) were recruited. Each participant searched 8 topics in total (4 for each version of the system) from a pool of 10 carefully selected search topic pairs.
Data collection technique(s) (Interviews, questionnaires, focus groups, etc): This study employed user-oriented evaluation methods for information retrieval systems. The participants filled out a searcher background questionnaire before the search assignment. After a brief training session, they were assigned to one of the arranged experimental conditions. They completed a search perception questionnaire and were asked to indicate the relevance of two pre-judged documents after each search. A brief interview was conducted when they finished all search tasks. Search logs with search terms and ranked retrieved documents were recorded. Effectiveness and efficiency of retrieval were based on the relevance judgments provided by TREC and the time spent.
Results: The important findings are 1) displayed MeSH terms are most useful for domain experts in terms of precision measure; 2) the two versions of the experimental system can produce comparable search results; 3) there is a significant interaction effect between versions of a system and searcher’s level of search experience in terms of precision; 4) perceived usefulness of displayed MeSH terms does not correlate with the search outcome, while terms in the Non-MeSH version do; 5) user perceptions of search task difficulty positively correlate with the time spent. The results advance our understanding of the usefulness of displayed terms and the relationship between user perception and search outcome in interactive information retrieval systems.
The importance of narrative as a method of communication has been extensively discussed in social linguistics. Researchers believe that narrative allows people to create rich mental images and transcend the meanings of their experiences. Recently, narrative and visual tools have come up in the work of researchers in online interface design, who suggest that narrative allows the effect of social context inherent in it to provide qualitative learning experience, and they believe that visual tools also positively affect the mental activities of the individual. Though these have important implications to instructional design, the inquiry of narrative, its relation with imagery, as well as the effects of using visual tools have been limited. Especially, the objects and the methods of investigation of the relationships between narrative and imagery are unclear in the current literature. In addition, the case-by-case style of narrative study has also been criticized as being more art than research. With this situation, seeking “a logical construction that accounts for narrative’s difference from other text-types” is expected. The main objectives of the dissertation study are: 1) to examine the functions of narrative and mental imagery in cognition for the purpose of discovering the inner relationships among them; 2) to examine how they could be interpreted when different sign systems (semiotic resources) and internal and external representations are concerned with the construction of meaning. The former objective is critical to the empirical investigation of narrative, whereas the understanding of visual representation has important implications for context design, automatic graphics generation and image retrieval. This study, using a general cognitive method, attains the objectives also by proposing a framework interpreting the functions of linguistic and visual resources and empirically verifying the established objects through a contrast between an online course in narration and in expository text. Combining qualitative and quantitative approaches, the empirical studies collected data from questionnaires, observation, test scores and open-ended essay questions with a sample of more than 150 library professionals. This poster will focus on presenting the established objects of investigation and some key findings from the ongoing data analysis.