As Open Access Is Public Access, Can Journals Help Policymakers Read Research?

John Willinsky (University of British Columbia)

Abstract: A study was conducted with 13 federal policymakers in Canada of a "Research Support Tool" intended to assist readers of open access research articles online to build a context for their reading. Interviews were conducted as the policymakers were introduced to this tool that would form part of the publishing environment for journal articles. The study found a growing interest among policymakers in the use of online and especially open access research. The tool proved helpful in providing access to author background, related studies, and relevant media stories, but posed real challenges to the users in design, intent, and the fairness of the databases it drew on in offering readers a greater context for the article they were reading.

Résumé : Une étude a été menée auprès de treize stratèges fédéraux au Canada sur un « Outil d'appui à la recherche » conçu pour aider les lecteurs d'articles de recherche interrogeables en ligne à élargir le contexte de leurs lectures. Des entretiens ont été menés avec ces stratèges quand on leur a présenté cet outil, qui servirait à compléter les articles savants édités en ligne. L'étude a observé un intérêt croissant parmi les stratèges pour la recherche diffusée en ligne, surtout celle à libre accès. L'outil s'est avéré utile pour accéder à des études connexes, des articles pertinents dans les médias généralistes et des informations sur les auteurs. En revanche, elle a posé de véritables défis aux utilisateurs par sa conception et son orientation, ainsi que par son choix de bases de données servant à élargir le contexte des articles que lisent les lecteurs.


During the past few years, a new publishing model for scholarly journals - one that makes the contents of online journals and e-print archives free to readers - has established itself as an idea to reckon with. Although this new model, known as open access, is being deployed in various forms by a relatively small number of journals and archives in the academic world, it is proving itself a viable alternative to the growing corporate concentration in journal publishing that has resulted in increasingly expensive subscription fees and a reduction in the circulation of knowledge. This open access approach is being hotly discussed in parliamentary committee hearings on scientific publishing in the United Kingdom, and the topic made its way into the "top science stories" of 2003 listed by Nature, Science, The Scientist, and the Wall Street Journal (Suber, 2004).

At this point, open access takes a wide variety of forms. It is being used by authors who post published articles, otherwise available only to subscribers, in open access e-print archives, and it is being deployed by journals that offer open access six months after initial publication for subscribers, as is the case with the New England Journal of Medicine.1 Whatever the form, this idea that people have a right to this research and that greater circulation of this knowledge is good for research and for researchers is causing people to rethink the future of the scholarly journal.

Most discussions of open access focus on opening access for other researchers, whether working in developing countries or at institutions that have had to reduce their subscriptions over the years due to the increasing cost of journals. As a result great gains have been made, for example, in providing open access to medical and agricultural research literature, through the efforts of the World Health Organization, for developing countries. Yet what is seldom considered in all this talk about open access as a means of increasing access to knowledge is the simple fact that open access is public access.

The significance of public access to research has already been felt in the field of health, contributing to what the Pew Foundation calls an "online health revolution" and leading to what some doctors are calling "shared decision making" as they confront patients holding printouts of online research (Brownlee, 2003, p. 54; Murray et al., 2003). While I discuss the impact of this public access in greater detail elsewhere (Willinsky, in press), in this article I want to consider the responsibility of and opportunity for the research community to support the potential of much greater public access to research and scholarship. More specifically, I report on our work through the federally funded Public Knowledge Project at the University of British Columbia with a small group of 13 policymakers in assessing an innovative online publishing environment intended to assist them and other members of the public in reading, interpreting, and utilizing research in a thoughtful way. Policymakers are but one set of potential readers of open access research, we recognize, and we plan to work with a wide range of readers to better understand whether - and if so, how - this public reading of academic work can be better supported, extended, and enhanced.

This research with policymakers and reading is part of a larger effort, by the Public Knowledge Project at the University of British Columbia, to develop and test publishing systems that support open access for research and scholarship. For example, the Public Knowledge Project has released an open source journal management and publishing system, Open Journals Systems, that is not only free to download, but reduces traditional publishing costs to the point where open access may be economically viable for scholarly societies and other independent journal publishers. The project also seeks to promote open access by improving the scholarly and public quality of the research published by open access journals, and one means of doing that is to enhance the reading experience. The research may be in the reader's own field or the point of a passing interest or of some professional or personal concern. The idea behind this study was to place the reader of open access research within reach of a good (open access) research library that offered a richness of context for such reading, while trying to ensure that the research was connected to a larger world of information.

Offering readers a greater context for their reading was our constant theme. In the study of reading, context has been identified as precisely what a competent reader brings to bear on their reading, and context - as a way of bringing relevant ideas into relation with the work being read - was something that Internet technologies could support. We also sought, in terms of the context in which an article is presented, to make explicit what was normally assumed or implied, from whether the article had been peer-reviewed to just what those peer-review policies might be.2

How can an online publishing system provide readers with a rich context for reading a research article - a context for experienced and novice readers, for those critically engaging with the very claims of the article as well as for those who lack a context for making sense of key ideas in the piece? Although online systems have been developed to connect the references an author cites to the original sources, if they are also online, we thought it was important for the journal itself to provide readers with a broader context, one that is automatically maintained and updated, one that would allow readers to make connections that went beyond those selected by the author.3

A research support tool

What we came up with was a linking device or support tool that would sit in the margins of a research article published in online journals. It would provide readers with a set of links to relevant resources, based on discipline, and each link would use the specific metadata (in the form of keywords) for the article to guide readers in locating materials that provide a richer context for reading the article, with the option of drawing on materials from open access research, instructional, media, and government databases. We have named the device the Research Support Tool (RST) and have made it available with Public Knowledge Project's Open Journal Systems (OJS) and Open Conference Systems (OCS), with plans to develop it as a free-standing tool that can be employed with other journal publishing and management systems.4

The RST sits to the side, in the margins of the research article, looking much like a bookmark. It offers readers 10 to 15 links, depending on the disciplinary version selected (Figure 1).5 (For a working demonstration of the RST, see The first thing that the RST does, after identifying itself, is introduce the piece the reader has called up. The RST identifies just what the text is - article, book review, editorial, and so on - and what its status is, indicating whether, for example, it is a peer-reviewed article or a non-refereed book review (with an explanation for these terms found by clicking on them). This introduction takes the form of "For this peer-reviewed article …," which is intended to indicate to readers that the links that follow are meant to serve the reading of this particular article. Each of the links that follows below has been chosen to provide another perspective, a further context, for reading the study. For example, View Metadata makes the article's indexing information, or metadata, readily available to readers (Figure 2). This provides a sense of what the author thinks the article is about with keywords, research method, and coverage, as well as who sponsored the work and what copyright restrictions exist.6

There are basic supports for readers, such as Define Terms, with which any word in the article can be double clicked, leading to its definition in an online dictionary (with two choices of dictionary sites provided, as a scholarly check against lexicographical biases). And there are more demanding resources, such as Other Works (by the author), Related Studies, and Related Theory, which encourage readers to compare the article they are reading to related works, whether to critically position its claims, for expert readers, or to gain a stronger sense of what is meant by one of the article's primary concepts, for not-so-expert readers. To ensure that only relevant materials are offered in building a context, rather than overwhelming the reader, we have employed a relatively simple trick. When a reader clicks on an RST link, such as Related Studies, up pops a window with the two search terms (the author's keywords) that can be fed, in a Boolean search, into any of the freely available full-text or abstract databases that have been provided with the RST, which can be modified by the journal's editor (Figure 3).7

With a click on "Search" in front of each database, the author's terms are fed into the search engine of research databases selected (with an information link on each database provided). This produces, in the case of Related Studies, a list of related studies that the reader can use as points of comparison or studies to pursue in themselves. A similar process takes place with the other links on the RST, allowing the reader to drop in on a forum discussing the topic or read into the theory behind the study. The reader can learn more about the available databases that provide a context for the study as well as change the search terms to customize the search within the databases. Readers are also led to see that the context for reading research is not always other research, especially in a field like education. The RST uses similar search principles to scour the databases listed under Government Policy and Media Reports. The Instructional link checks for teaching materials on the subject. Through the RST, readers can also make comments on the paper and e-mail the author or a colleague.

Having set out the idea and design behind the RST, we are left with the question of how it works and whether readers will use it. We also want to know whether it can contribute to greater engagement with the research that will augment its scholarly and public value. For this research stands poised to become far more universally available through the Internet, if the academic community can begin to see the rightness and value of such greater circulation. In an initial testing of the hypothesis that lies behind the development of the RST, we shared a version of it with a group of policy officials working in the federal government of Canada. These are both professional consumers of research and non-expert readers. They speak very much to the public impact of research, and our earlier research with this community had established that they have been deeply affected in their research strategies by online sources of information, with a strong dependency on open access sources of research (Willinsky, 2003c).

Policy officials research online

A member of the Public Knowledge Project, Michael Sutton, met individually with 13 policy officials in the Canadian government. He asked them about their daily use of the Internet as a source of information and he took them through a working model of the RST. Our goal was to better understand the actual and potential role in the policymaking process of academic research available online, as well as to see whether a tool such as the RST would support these individuals' work with research. The sample included a senior policy advisor and a division head for Human Resources Development Canada, a principal researcher and policy analyst for the Bank of Canada, a program analyst, and a number of officials associated with the government's Policy Research Initiative, as well as officials with Ministerial Services, the Department of Justice, and the Public Policy Forum (Table 1).

Table 1: Policymakers participating in review of Research Support Tool
No. Agency Acronym
4 Human Resources Development Canada HRDC
3 Bank of Canada BoC
2 Policy Research Initiative PRI
1 Ministerial Services MS
1 Department of Justice DoJ
1 Public Policy Forum PPF
1 Canadian Policy Research Networks CPRN

In the course of the interviews, these policy officials made it clear that they rely almost exclusively on the Web for gathering information and research, with the figure 90-95% reliance on the Web coming up more than once: "Internet is now my starting point for all my research" (BoC-1); "I use the Internet almost exclusively" (PPF-1), and "Right now I do all of my work exclusively on the Internet. I don't have time to go to books or other sources" (PPF-2). When asked about their usage during the previous workday, reference was made to using the Internet 10-15 times that day. Still two people in the sample, one a directorate head and the other an associate in the Policy Research Initiative, said they only used the Web half the time in seeking information (HRDC-3; PRI-1), while a third person, working with the Public Research Initiative, spoke of doing "a lot of my work in hard copy, especially things that are not articles, like books on history," though "I do have to supplement this with some work on the Internet" (PRI-2). Those three examples notwithstanding, the typical policymaker is moving through a world of online information.

We also asked the policymakers whether they were now consulting a wider range of resources on the Internet than they were two years ago, when we did our last study with this group (Willinsky, 2003c): "Significantly" one official responded, "I'm using far more international comparison knowledge because it is readily available. It is all well and good to see a citation to a London, U.K., white paper on knowledge pools, but getting your hands on it has been a challenge. Now that those papers are available, yes, I use them extensively. OECD [the Organisation for Economic Co-operation and Development] is a prime example. It used to be that using OECD [was] painful, aggravating, and sometimes [a] painful experience - it is now a no-brainer" (PRI-1). The same subject also spoke to how the cronyism of pre-Internet days - in which information was gathered by chatting up an academic contact or acquaintance - is still in place, but this traditional method is no longer the end point, by any means:

First I go and chat with some professor who's been through the wars and tells me exactly what I am looking for because very often what I think I am looking for is not really what I am looking for. And then I go check it out with the guy sitting next door to him, because he is the counterbalance point. Then I start to go hunting down. I tend to triangulate, and one of the things I will do is go back to the journals and I will try and find the summary articles. (PRI-1)

Another factor that appears to have changed over the past two years is the policymakers' confidence in the reliability of the information found on the Web. "I would say that the level of the trustworthiness has gone up," said one researcher, and yet as a caution this same policymaker in the Public Policy Forum had this to say about online journals: "I don't use them" (PPF-2). On the other hand, one senior researcher responded to our question about how he had used the Web as a source of information on the previous day in a way that suggested that providing open access to journal articles was making a difference in the impact that research was having on the policy process: "In the last day or two I used the Web to track down information on a specific European workplace survey conducted in the late 1990s. I had a reference to some articles by a specific author and was able to find his website. Links to relevant articles were available on his website and I obtained the articles I needed online. Moreover, exact wording of some of the survey questions in which I was interested were cited in an appendix" (CPRN-1).

These policy officials did display an indirect enthusiasm for open access to research. That is, they tended to avoid subscription and pay-per-view sites: "I often go to sites and I say, 'Why do I have to pay for this?' " (HRDC-1); "No [I don't use them] - because I can find the information elsewhere" (PPF-1); "I find [toll sites] really annoying" (BoC-2); and "I always found it annoying to track down an article only to find I had to pay for it if I wanted it… Online subscriptions can be very expensive" (BoC-3). However, the Bank of Canada appears to provide its employees with a wide range of subscription services covering the economic literature. Others had found work-arounds in their online searches for research: "I will usually contact the library for them to get a photocopy as an alternative to paying for an article" (HRDC-4). Ultimately, research that appears in open access journals or is available through authors self-archiving is in a position to exercise a far greater influence on policymaking processes, if only in terms of timing of access: "These [pay sites] are so frustrating to use. If I have to go through lengthy processes to get it within tight timelines I will not use it" (PRI-2).

There was a sense of entitlement, if not all that well worked out, among these professional users of online information. "You start having to pay for them and that to me is a concern because one of the greatest advantages of the Internet is being part of the 'proletarianization' (if that is a word) of information, of sources of data" (PPF-2). And yet in at least one case, an analyst from the government's Policy Research Initiative had the economic issues well worked out:

If you look at large corporate interests that manage our knowledge dissemination and distribution, they have structured their interaction and participation in such a way that they are "rent seeking": providing a service based on a critical mass (monopoly). E-journals are quickly getting around this. The problem is they are so fractured and organic and distributed that there is not that one critical mass you would have with any big organization to manage or market and push that interest. As we start to see these bottom-up, self-organizing networks that start to enable you to do detailed contextualized information linking and information dissemination using a common protocol, the value added is going to happen very quickly (tipping point). Right now e-journals are not at that point. (PRI-1)

If these policymakers are given to going online to find research in such a daily and consistent way, with a strong predilection for open access research among those whose units did not have substantial online subscriptions, the question for those of us interested in the public presence of research is, then, whether there are ways of supporting these potential users' ability to comprehend, interpret, and utilize this research. The first step in that process was to see what sense a small sample of policymakers would make of a tool designed to do that. Does this initial approach to providing a suitable context make sense to them; do the categories and sources we have chosen add to confidence with which they turn to online research? The subjects were asked to explore the functionality of the RST, which was presented in the online environment of our Open Journal Systems Demo Journal with the research article "Understanding in the Absence of Meaning: Coming of Age Narratives of the Holocaust," by my colleague Theresa Rogers. The subjects were given a scenario for reading the article in relation to a policy task.8

Policy officials use the Research Support Tool

The RST provoked a degree of both confusion and appreciation on the part of the policymakers in their first encounter with it. Their reaction attested to the challenges faced in trying to create a device that will assist a wider audience than typically encounters research, or as one program analyst put it, in regard to the Capture Cite feature, "I think part of what's going on here is we non-academics [are] trying to use something we do not usually use" (HRDC-1). Even the idea of what an RST might be was not immediately clear, and given the intended innovativeness of the tool, it is hardly surprising that just what it might be about was not readily apparent, or as one subject explained: "I am assuming that the Research Support Tool for this peer-reviewed article means that this is a tool that would be focusing on this particular content and … then I would get confused because I would say, 'How can it be a Research Support Tool?' " (HRDC-1). But then as he explored the RST a little further: "So if I click on it and find out what the heck it is I will be pleasantly surprised" (HRDC-1). The policymakers had suggestions intended to address the confusion that the RST was capable of engendering: "If [the RST] has said, 'This is a Peer-Reviewed Article,' or 'This particular article was peer-reviewed,' instead of 'For this peer-reviewed article,' then it might have made better sense" (HRDC-4). The explicit identification of peer review status of the article did not prove to be a notable point among the 13 officials, with only one exception: " 'For this non-refereed article' - it gets you to the status or the definition; this is not just a piece of fluff" (PPF-1). Far more commonly, the author's institutional affiliation proved to be a much greater indicator of quality assurance for these users.

With the View Metadata link on the RST, there was a degree of recognition of this technical term itself among a minority of policymakers as well as support for its usefulness - "Very useful for a library search" (BoC-1) and "Could be useful in helping me refine a search" (PRI-2) - and a sense of its value as an educational end in itself: "Yes, it is really kind of like looking under the hood" (HRDC-1); and "This is helpful. It captures in one place all the kinds of questions you might have about the reference … and about the authority of this selection - given at a conference, published in a journal" (HRDC-4). Yet concerns were also expressed over the use of "metadata" as a form of unnecessary jargon: "Why say 'metadata' when you can say 'Tombstone data' … that would be more descriptive … If you think of people whose native language is not English, you might just simplify it for them, too" (MS-1). As if to confirm the point, at least two of the policy officials mistook metadata for meta-analysis: "I think of this in terms of meta-analysis, where people are pulling together quantitative studies from a whole wide range of studies and then doing a meta-analysis" (CPRN-1). This would not be the only time we could see a divide between educating the readership about research and online systems ("metadata" is a term used with online documents and files of all sorts), and the need for simplicity and support.

There was also some confusion over the use of the short form Supp. Files for "Supplementary Files," which is used to provide researchers with an easy way to post their data sets, research instruments, and perhaps unpublished documents. The officials raised interesting points about what exactly supplements a research article and whether appendix would have been a better term, but they did appreciate the general concept and value of including more about the data used in the research: "…supplementary or support files is what I would anticipate when I click on that. Now for this paper, this article, I would expect more research data; the research control groups and those types of things. That is the kind of information I could use to be able to view the actual data. Where was the study conducted? The ages, number of students - those kinds of things" (HRDC-1).

One feature of the RST that did make immediate sense was the ability to double click on any word in the research article, in order for that word to be sent to one of two online dictionaries for a definition (Define Terms): "Wow, that's powerful; that is kind of like More Info, (context sensitive) … After you have used it once, you will definitely want to go back again. It is a very powerful tool" (HRDC-1) and "That's excellent - does it always go to the same Web dictionary …?" (PPF-2). Yet even here, the feature was seen as limited by not being linked to subject-specific or specialized dictionaries that might pertain to the technical language used in the article. There was also the sense of how this tool served certain readers at certain times better than others: "If you are getting into a totally new area, there may be more uses for it" (BoC-3), as well as an awareness of biases, even among the two dictionary choices presented: "And we're using an American dictionary?" (DoJ-1).

Among all of the links provided by the RST, however, the one that clearly provided the most valuable context for evaluating the research article proved to be the Author Bio. The example used, with Theresa Rogers, was perfunctory, and many of the subjects felt there should have been more information available in this example, making it clear in the process just how important this feature was: "I would expect to see a little more robust … information … like e-mail link, website link, her credentials, and affiliations … Is this the first article they have ever done and are they a subject specialist in this area?" (HRDC-1); "Would want a longer biography than what is there; a good paragraph or more saying this is where they came from, this is their research field; this is their expertise" (PPF-1); "We should see key articles, key publications, corporate websites that would direct you to her or some of her academics. Some actually attach their CV" (DoJ-1). In a similar vein, at least one policymaker saw the value of locating the author's position through the use of the (author's) Other Works feature of the RST: "In that particular repository - the thing that I see right here is that the author is definitely coming out of a pedagogical stream and she is working with Andrew Schofield" (PRI-1).

The interest in finding a basis for trusting a work was clearly there, and for non-experts this trust has to be about credentials rather than a quality they can otherwise detect or judge within the paper itself. As one policy official put it bluntly: "Anybody can disseminate material today … affiliations become everything now. Like the way we judge the type of people we get in … as speakers. It's incredible how many come from two institutions: Yale and Harvard because of the name - it communicates so much authority, even more so than it did in the past" (PRI-2). This official made it clear that access had its costs, as more information led to less general credibility for materials found online. And yet to prevent a complete narrowing of the field of expertise to two or a handful of institutions, it seems clear that additional measures of value and credibility need to be introduced into the system of online publishing, including perhaps citation counts for an article.

A further means of judging the claims of the article itself was to see the article in relation to other studies, a task more suited to experienced readers of this literature. Here there was but one official who held to the value of Related Studies: "Certainly could be very useful … it is the most appropriate one [of the RST links]. Although … citations … is the way I initially go about finding related studies … unless there are more recent ones … and that is where this feature would be more useful … finding the stuff that has been most recently published" (BoC-1). By the same token there was little appreciation of why it would be valuable to see the ideas in the research article discussed in online forums, against our expectation that inexperienced readers of research would find the informality of the online forum a friendly enough format, as well as appreciating the controversy and back-and-forth quality of the forum: "I have never really seen these in the context of articles, per se" (BoC-2) or "I tend not to use that sort of thing but if I did it would be helpful" (PPF-1). On the other hand, the RST's technical ability to locate a relevant forum for the research article was found impressive in at least one instance: "Did it get that, [the results], from the keywords? [Interviewer: Yes] … that is pretty good" (BoC-3).

We were a little surprised that the RST's Instructional links, intended for those who had teaching responsibilities, turned out to pertain to those working in the federal government as well: "I am quite curious of these resources because when I go on these technical missions [for the Bank of Canada], I just got back from Albania where I taught for two weeks, and a lot of the material I used [for my instruction] was found at the Learning and Teaching Support Network Centre for Economics" (BoC-2); and "I have worked with some lesson plan repositories" (PRI-1).

In working with policymakers, we were especially interested in their response to the RST's Gov. Policy, which enabled them to search Canadian and American government sites for documents related to the research article: "This should be interesting" (MS-1) and "So this is an interesting database….What you are doing is taking us into other search tools, which I think is quite valuable and no one search tool is going to be everything to everyone…I was thinking that eventually people would develop their own trail/route through this" (HRDC-1). By the time the officials had reached Gov. Policy, they had worked through the better part of the RST and had begun to figure out how it was taking advantage of other related resources to provide a richer context for reading any given article.

Another area that received wide recognition as helpful to connecting the research article to the larger world was the Media Report link: "As a public policy developer and researcher, this functionality is very important" (HRDC-4); "Often I would go through the Globe & Mail website separately, and having the connection of the Research Support Tool (RST) would be quite useful" (BoC-1); and "[This feature] could be useful to find out if the issue is being debated"(BoC-3). A number of officials expressed concern with bias in the selection and ordering of the media represented in the list presented, pointing to the active part to be played by readers in ensuring the fairness of the connections and opportunities: "Where it would bother me is if they were all particular media sources of a particular political perspective (the right versus left), but this looks like a pretty mixed bag" (PPF-1). I should add that the list of media sources, as well as their order, can be altered by the editor of the journal, and is not hard-wired into the RST. It seemed that the more important a contextual element, from the author biography to media reports, the more adamantly the policymakers wanted in to fix it, to make it fair and representative, complete and logical, too.

Apart from evaluating the particular links and resources provided with the RST, we were also interested in whether the organizing idea of providing a set of "context" links made sense. The use of the actual term in RST clearly did not work well, as it was not clear what was meant by context without a context: "Context is meaningful, but is it the context of the article, author, or where it fits in the literature?" (BoC-2); "The word 'context' doesn't mean much" (PPF-1). There was much confusion over whether Context was a link like the others or a subtitle for the other links, clearly a graphic design problem. But for some it worked just fine - "My initial impression is again other relevant literature - how is this article situated in terms of ideas or research or other articles, and so on" (CPRN-1) - and for others, the use of "context" became more apparent after they had used the RST for a while: "For me the context information was more useful. And to see what other works this author has produced…is [this] a one-time article and what else has this person done" (HRDC-1) and "For me it would help me position the article. Where does it rank with competitors or what else has she [Rogers] done?" (BoC-2).

Another more general question about how to best support the reading of research online revolved around how explicit to be about the sources and databases the RST was set up to consult. Our original choice had been to spell out the details, presenting the names of the databases as well as listing the search terms, which was seen as both more educational and scholarly. It turned out to be generally appreciated by the policy officials: "I like the idea of listing what you've got" (MS-1) and "OK, I can search these…26 journals and I can change the terms for searching … That's great" (HRDC-4). The diversity of links also received at least one favourable note: "Certainly, as a researcher you have to try to cast a wide net of research" (BoC-1). The RST also provided a "more information" icon with each database or resource provided (which linked to an "About" page for the item), and this was appreciated by at least one policy analyst: "So for every content link I always say - and here's the link where this came from, or the originating organization or at least one level up so that people can retrace if something like that happens" (HRDC-1). The journal itself, in which Theresa Rogers' research article appears, also had an About link at the top of the page, which linked back to the journal's editorial policies and personnel: "I would want to know who was on that journal committee or peer community" (PRI-1). Not surprisingly, at least one official would have preferred some blurring of the distinctions in the name of efficiency: "I would say the e-journals, related theory, related studies - you could probably mush those together quite easily" (PPF-2).

Another interesting phenomenon we discovered in the course of conducting this study is that once people get the hang of the tool, they expect the tool to do more than it reasonably can. For example, one policy analyst felt it would have been better if all the available databases were searched prior to his use of the RST, so that it would only present to the reader those that had valid and pertinent items: "Because it's not doing what I am assuming it's doing, it's not pre-testing, it's just saying, 'Oh, there is this search engine here and we'll be nice and we'll give you the search engine as if you had opened the search engine and put the word in and hit search' - I don't want you to do all that work for me if there is nothing at the end of it because you are forcing through those screens" (HRDC-1). Or where there was a limited number of databases available, they were ready for more: "I would want more choices on theoretical repositories" (DoJ-1); and, "I'm guessing I would be looking for works and different banks of theories that relate to these kinds of principals…I think you would like to have as many as possible" (PPF-2). We took this as an encouraging sign that these discerning researchers and information consumers would be able to further exercise their skills with the RST. The "ranking" of journals was another feature mentioned, speaking to how easily the stakes are raised with research expectations among a professional group like this: "If I did not know the field, then it would have to be ranked or it wouldn't be too useful" (BoC-3). The ranking of journals and articles, based on citation levels, is commercially available through the ISI Web of Science. However, an open access version of the citation index is under development, with Citebase (, which could eventually be incorporated into an RST-like tool.

The best summary of what was most valuable about the RST was provided by a program research assistant: "The way I would rate it is what is going to be most useful. The first is Author Bio, second thing - Other Works, third thing is going to be Online Journals and Similar Articles, then Theory, then Studies, then Articles I have to Pay for" (PPF-1). There was also a sense of having to trip through the RST that first time, amid confusions, misunderstandings, and second guesses, before really gaining a sense of what it is about: "Once you are through it [the functions] the first time, they will become pretty obvious" (ABC-1). Others saw advantages in it being a distinctly academic tool, designed to assist with the reading of research: "I like it because it is more academic. In a Google search you are going to get anything that is out there. And then you are going to have to sift through the 110 hits until you can find those that are more academic versus so many posting an advertisement or a media article" (DoJ-1).


Based on the contributions of this small sample of policy officials, we have some distance to go in improving the basic design of the RST. That is, improvements need to be made to a number of features before we begin to presume to test the effect of such a device on reading comprehension, the ability to evaluate the relative merits of a study, and whether it inspires greater confidence in using this as a research tool - with a need to test these qualities with a wide range of readers. Otherwise, confusion arising from the design of the tool becomes an unnecessary distraction in assessing the value of providing a richer context for reading research articles. Given the range of experience that readers bring, which was amply apparent with even a small sample of users, there are clearly limits to how well all readers can be served by such a tool. We are led by this work back to the proverbial drawing board, before returning to this and other communities of potential users.

One encouraging result of this study was its demonstration of the sheer appetite of policy officials for online research. They made apparent the considerable impact on policy processes and decisions by that large body of research produced by universities and research institutes, often at public expense. Now, it should be made clear, in terms of the larger argument for the public value of opening access to research, that simply having research play a greater role in policy does not necessarily improve the democratic quality of our lives, as I have argued elsewhere (Willinsky, 2002). There is an equal need to have the careful reasoning and painstaking data collecting of research, not to mention its diversity of approaches, values, and viewpoints, made available to public engagement and discourse on a larger scale.

This first point is, of course, independent of the work represented by the RST and our efforts to improve the value of the research for readers. However, this study did demonstrate that among the policymakers who participated there were various points of interest and support for this method providing greater support for the reading of research by professional and lay audiences. It was not love - or complete understanding - at first sight. But that would be a little much to ask for something with which these readers were not familiar with, in the already unfamiliar realm of academic research. Yet whether one considers the importance that the policymakers placed on knowing the author's background or their keenness for a fair representation of media coverage in relation to the research article, there are grounds for thinking that the publishing of research online can incorporate devices and tools intended to support this wider readership without significantly adding to the burden or expense of publishing. We intend to continue exploring questions of the value of public access to all areas of research, and whether that value can be increased by providing greater support for readers.

It seems important, at this point, for at least some of those involved in the online transformation of scholarly publishing to consider whether a much wider readership can be supported, in finding value and interest in this realm of knowledge. It is no less important to explore, as well, the means of providing a more critical and engaged reading by faculty and students. This is only to say that it will not be enough to simply move journals online, as if they had simply changed shelves in the great university libraries. This new medium needs to be tested in its possibilities for enhancing the scholarly engagement among a wide range of readers. In thinking about publishing models that might increase access to this literature, it would certainly make sense to take into account the publishing system's ability to support a greater number and range of potential readers. The promising development of open access publishing, in all of its various forms, from e-prints to open access journals, should be seen as not only reversing a decline in access to the research literature, but also as opening the research literature to both a larger public and the global scholarly community. The design of a better reading environment for this new form of online access is the necessary complement of this open access approach.


Both the Research Support Tool and Open Journal Systems have been developed by the Public Knowledge Project ( with the generous support of the Social Sciences and Humanities Research Council, the Max Bell Foundation, and the John D. and Catherine T. MacArthur Foundation. I would also like to thank the journal reviewers for this article for helpful suggestions, as well as Anne White.


  1. See Willinsky (2003a) on what are currently nine flavours of open access publishing. For the advantages of "self-archiving" through e-print archives, over relying on journals to go open access, see Stevan Harnad (2003).

  2. The intent is to provide readers with support for what Murphy & Alexander identify, in their Model for Domain Leaning, as the reader's "strategic processing" knowledge (2002). This knowledge is "invoked to overcome perceived deficits in understanding or to circumvent potential barriers to learning" and might include, in this case, an ability to review the article's metadata, to tap into online critical discussions of the article's findings, and to consult media coverage of the article's topic (p. 199).

  3. Crossref is "a collaborative reference linking service that allows the user to click on a citation and be taken directly to the target content" (; SFX "offers libraries a proven solution for linking their diverse, ever increasing set of electronic resources" ( Those working with open access publishing are working on ways of linking an article's references back to the work cited (Hitchcock et al., 2002).

  4. The Research Support Tool (RST) is currently available in 19 areas from agriculture to the social sciences, and can be added to and updated by the journal editor. Because the RST relies on open access research databases, such as in physics, it can only be as specific to a discipline as resources permit. For example, with few such resources in the Humanities, there is only one version of the RST available for this field, while in the sciences specialized versions of the RST exist in medicine, physics, astronomy, and other disciplines. See the PKP website, for demonstrations of OJS and the RST as well as examples of journals using these tools. It should be noted, as well, that the online journals hosted by HighWire Press ( as well as the "LinkOut" service of the PubMed index ( incorporate some of the same features as the RST, with the RST looking to other domains of knowledge, such as the media and policy, in seeking to provide readers with a richer context.

  5. This section of the paper, presenting a description of the RST, draws on Willinsky (2003b). The RST currently operates with both PKP Open Journal Systems and Open Conference Systems.

  6. Literacy research has identified how inexperienced readers have difficulties identifying the author's main idea - separating core ideas from the noise - which should guide the search for related points and arguments (Alexander, Pate, Kulikowich, Farrell, & Wright, 1989).

  7. On the value of demonstrating to readers the use of multiple search terms (that is, a Boolean search), see Vine (2001).

  8. The scenario for the walk-through was as follows: "You are working in a Ministerial office, and the matter is policy on teaching the Holocaust. You have been asked to identify the relevant issues, especially in relation to teaching literature dealing with the Holocaust in middle and high schools. The goal is to prepare an initial draft of a briefing paper, as well as identify resources for further committee work on this. You then come across this article by Theresa Rogers…"


Alexander, P. A., Pate, P. E., Kulikowich, J. M., Farrell, D. M., & Wright, N. L. (1989). Domain-specific and strategic knowledge: Effects of training students of differing ages or competence levels. Learning and Individual Differences, 1, 283-325.

Brownlee, Shannon. (2003, March 16). The perils of prevention. New York Times Magazine, 52-55.

Harnad, S. (2003). Eprints: Electronic preprints and postprints. Encyclopedia of library and information science. New York: Marcel Dekker. URL: [September 28, 2003].

Hitchcock, S., Bergmark, D., Brody, T., Gutteridge, C., Carr, L., Hall, W., Lagoze, C., & Harnad, S. (2002). Open citation linking: The way forward. D-Lib, 8(10). Retrieved November 6, 2002, from URL: [September 28, 2003].

Murphy, P. K., & Alexander, P. A. (2002). What counts? The predictive powers of subject-matter knowledge, strategic processing, and interest in domain specific performance. The Journal of Experimental Education, 70(3), 197-214.

Murray, E., Lo, B., Pollack, L., Donelan, K., Catania, J., Lee, K., Zapert, K., & Turner, R. (2003). The impact of health information on the internet on health care and the physician-patient relationship: National U.S. survey among 1,050 U.S. physicians. Journal of Medical Internet Research, 5(3), e17. URL: [September 29, 2003].

Suber, P. (2004). SPARC Open Access Newsletter, 69. URL: [April 25, 2004].

Vine, R. (2001). Real people don't do Boolean: How to teach end users to find high quality information on the Internet. Information Outlook, 5(3), 16-23.

Willinsky, J. (2002). Education and democracy: The missing link may be ours. Harvard Educational Review, 72(3), 367-392. URL: [April 25, 2004].

Willinsky, J. (2003a). The nine flavors of open access publishing. Postgraduate Journal of Medicine, 49(3), 263-267. URL: [April 25, 2004].

Willinsky, J (2003b). Open access: Reading (research) in the Age of Information. In C. M. Fairbanks, J. Worthy, B. Maloch, J. V. Hoffman, and D. L. Schallert (Eds.), 51st National reading conference yearbook (pp. 32-46). Oak Creek, WI: National Reading Conference. URL: [April 25, 2004].

Willinsky, J. (2003c). Policymakers' use of online academic research. Education Policy Analysis Archives, 11(2). URL: [April 25, 2004].

Willinsky, J. (in press). Rights and vanities: The case for open access to research and scholarship. Cambridge, MA: MIT Press.

Appendix I - Interview Comments

Date of Interview: September 11, 2003.

Public Knowledge Project Policymaker Protocol, II

  1. Think-Aloud Protocol with the Research Support Tool

    1. Comments During Quick Browse of the Article

      • It's about different ways of looking at the experience of different Holocaust victims.

    2. Informant Reaction to Research Support Tool Interface Components

      1. < For this peer-reviewed article

        • This is a description of what this article could be…there are three different ways, the status and the type of the article.

        • I might highlight them or something to distinguish them from the other ones in the list. It didn't jump out at me that the number one which was on the list and was necessarily the one that this one is (peer-reviewed article).

      2. Capture Cite

        • Cite means citation.

        • What I think I'm seeing is what would be a formal citation for a bibliographic program - if you use this in academic context this would be a way to fairly easily capture bibliographic information of what you read which would be good.

      3. View Metadata

        • Generally speaking, Meta is sort of higher level or sort of broader data so for me this would be the general descriptors of the article as opposed to what the article itself.

        • I might do a 3(a) or 3(b) or something like that so people don't think that that there is some kind of replication of Subject - I guess it would depend on how important this is to aid this particular methodology, because you have Subject; Subject on one line - you might have Discipline; Discipline on the next or something like that. Just something so that people don't look at it and go, they made a mistake. The rest looks good.

        • I think having an author name and e-mail is a very good thing.

      4. Supp. Files

        • I would guess mainly things like appendices or sort of related files to the article.

      5. Print Version

        • It comes up with the reference and a good version of it.

      6. V Context

        • Context is a whole list of functions - there isn't an actual context function???

        • It is a little confusing that you can go for this peer reviewed article up here and click on that LEFT ARROW on context-or even a different colour for this box would be useful. I'm not sure how much strength the arrow adds in this case so maybe take the arrow off and put it (the actual text box) in a different colour.

      7. Author Bio

        • Just a short bio of the author.

        • If you had Noam Chomsky, I would expect a little bit of background and maybe some of the things that he has accomplished.

      8. Other Works

        • I might link this to the author bio especially in an academic context. Bios are often associated with lists of publications, other works, etc. - but other then that it looks good.

        • (The subject just opened archives OAI Harvester)

        • I am trying the education line; electronic text and education.

        • (Interviewer - it appears that the server is down right now.)

      9. Define Terms

        • That's excellent - does it always go to the same web dictionary...

        • So I went to the One Look dictionary search thing and I came up with 13 different definitions for the word cancerous.

      10. e-Journals

        • It is an electronic version of a journal that's in print or sometimes an actual journal that's only published electronically. I don't know if there is an actual protocol to it.

          (Interviewer: Do you expect it to have the full text or just an abstract of the article.)

        • I've seen both so again my expectations are limited on these kinds of things.

        • So I'll just go to the current issues and education as item #4.

          (Interviewer: The terms Holocaust and Young Adult Literature came from the metadata field).

        • I guess the question would be if there are more than two meta-terms, how does it choose more than two meta-terms. The other thing that you might want to do is list what the other meta-terms are.

      11. Related Theory

        • I'm guessing, but if there is a link - there are the two meta-search strings - I'm guessing I would be looking for works and different banks of theories that relate to these kinds of principals. I see that you only have one search engine there - education theory.

        • Would I expect to see more then one repository of theory to be searching - not necessarily - would I want to…probably. I think you would like to have as many as possible.

      12. Related Studies

        • This is the same deal as related theory.

      13. Pay-Per-View

        • My interpretation would probably be services that cost something to use.

        • I would say with any of these I think you would be well served to have a sentence or two right at the top that says - this function - you know - it is all fine and dandy to come up with something that you think is going to be sort of intuitive but I think you can also give people a hint at the top of the window as to what each function actually does.

      14. Online Forums

        • I would assume that this would be a list of online forums or search terms for online forums relating to...

      15. Instructional

        • I think Materials would say more then Instructional Materials.

        • I actually don't know a lot of these search engines.

      16. Gov Policy

        • I assume that what we are looking at here are government sites available. First Gov I would imagine is American?

      17. Media Reports

        • Is probably just like when you have databases for various media things (i.e., newspapers).

      18. Google Search

        • I'll click on it but I assume it just Google search.

      19. V Action

        • (No comment.)

      20. Email Author

        • I assume this is just sending the author an e-mail again populated by the meta-string.

      21. Email Others

        • If you are going to do this I think it would be useful to be able to attach the article; you might want to say that the URL is attached.

    3. Other Informants Opinions

      1. Would you read more if you had more time?

        • (No comment.)

      2. What has it contributed to your thinking about this particular article?

        • (No comment.)

      3. RST strengths, if any, and weaknesses?

        • I would say the e-journals, related theory, related studies - you could probably mush those together quite easily.

        • Again some of the search tools you might want to combine into groups and then break them out into the window but again that might be harder.

        • As I said I find it a little confusing that you can click on the title here (peer-reviewed article) but you can't click on the title for the other two (Context and Action).

        • The actual research support tool piece on top - I would actually put that in a different colour box or play it up with a bigger text or something. I think it would make it easier if you broke it up somewhat - that is a long string of text. Again, if you click on what changes the colour or whatever that would be useful as well because that way you would know which one you need to. Like if you click on Author Bio and you came back out of it if it was blue and the text was yellow or something.

      4. Are we correct in our assmumptions about how many times a year you might use a tool like this, or would it be used more than twice a year?

        • (No comment.)

      5. Miscellaneous Remarks

        • (No comment.)

  2. Initial Interview Questions on Internet and Information Use

    1. Compare your current use of the Internet to find relevant research in relation to other sources, such as print journal or books.

      • Right now I do all of my work exclusively on the Internet. I don't have time to go to books or other sources.

    2. In what ways have your choice of sources changed, if at all, over the last two years? Would you say that the role that outside sources of information play in the policy processes has changed at all over that period?

      • I think it is becoming much easier to rely on the Internet. You can get a lot more quality materials on the Internet. Like the same kinds of materials - journals, newspapers and other sort of primary sources.

      • I would say that the level of the trustworthiness has gone up. It is like anything else, you take a look at the body that's out there - some of them are going to be from sources you know, some of them are from reputable sources, some of them are not so reputable sources. It is like any other process that way. You have to be able to use your own discretion on what's a viable source.

    3. How often, if at all, did you consult Web resources for information purposes during the course of the previous working day? Give two examples that come readily to mind.

      • Probably 5 times.

      • I looked at websites - the Canadian Association of Broadcasters; looking at recent publications; going to Stats Canada looking for a daily grouping set.

    4. What are the different ways, if any, that you use to locate information that would qualify as "research" by your definition?

      • Generally a Google search and then once you go into different sites there is often links within other sites so it is a good tool. Also COPERNIC - it is a metadata searcher - you can use that as well if necessary.

    5. Can you name any good websites, portals, engines or other places that have proven to be reliable sources of relevant research?

      • Google and COPERNIC are probably my two that I use.

    6. Have you used any pay-per-view sources for research? Examples? Do you have access to any subscription-based online journals or sites? Examples?

      • Stats Can is a pay-per-view source but, frankly, I think the site should not be pay-per-view. It's a government service so it shouldn't be a cost recovery service.

      • On line journals - I don't use them.

    7. Can you give any examples of "free" or open access sites or journals, which you use for research?

      • Again, I don't go to forums; I don't go to open sites. I tend to look specifically for a piece of data or a certain item.

    8. Do you have comments or concerns about the current or changing state of access to research and related forms of knowledge relevant for policy work?

      • It is like anything else. Those things get more stylized and more formalized - you start having to pay for them and that to me is a concern because one of the greatest advantage of the Internet is being part of the "prolitarianization" (if that is a word) of information, of sources of data. So I think you are seeing sort of this re-entrenching of a digital publishing industry base. People try to cost recover and recoup their investments and things like that.

  •  Announcements
    Atom logo
    RSS2 logo
    RSS1 logo
  •  Current Issue
    Atom logo
    RSS2 logo
    RSS1 logo
  •  Thesis Abstracts
    Atom logo
    RSS2 logo
    RSS1 logo

We wish to acknowledge the financial support of the Social Sciences and Humanities Research Council for their financial support through theAid to Scholarly Journals Program.