Yearly Archives: 2009

Preparing more people for a diverse future

The TED Talks lecture series is a wonderful intellectual and cultural resource. I’ve been a fan for years and on this long weekend relished the opportunity to catch up on some I’ve had in my queue. One of my favorites is by Ken Robinson, because it highlights both a goal and a challenge of my research.

First, the challenge:

I have an interest in education — actually, what I find is everybody has an interest in education. Don’t you? I find this very interesting. If you’re at a dinner party, and you say you work in education — actually, you’re not often at dinner parties, frankly, if you work in education. (Laughter) You’re not asked. And you’re never asked back, curiously. That’s strange to me. But if you are, and you say to somebody, you know, they say, “What do you do?”and you say you work in education, you can see the blood run from their face. They’re like, “Oh my God,” you know, “Why me? My one night out all week.” (Laughter)

Education, to many, is a very dry topic. That’s because, for many, the practice of education is very dry. If you ask a space alien what education is for, he posits, they’ll say it’s training to be a university professor. That’s the pinnacle of the process, right? But it takes much more than university professors to make the world go round, of course. How can our system of education support the diversity of needs and foster the diversity of talents of a rich and dynamic society? That’s the goal.

I ask “how” not “if” because I don’t see it as a choice. Globalization, mechanization, intelligent machines… these demand that we develop human resources, fostering the gifts of each person however they may manifest. Motivated by the example of Gillian Lynne (starting 2:30 in the talk), Robinson argues that we increase the amount of education in the arts, even specifically teaching dance to everyone. Such sentiments underestimate the constraints on time, money and attention for and of students. I would imagine Robinson is not a fan of intelligent tutoring systems, how they involve sitting at a desk and developing, most often, a laser focused set of skills.

But I think research in intelligent tutoring systems and other computer supported learning can help address the diversity he speaks of and even help students learn to dance, if that’s something a community values. Firstly, they can help increase the efficiency with which “maths” (he’s British) and other “left brain” skills are learned, leaving more time for other enrichment.  Secondly, and this is a goal of my research, computers can help personalize instruction to each student’s needs and interests. The computer can devote its unblinking attention to the individual, drawing on a wealth of data about their prior interactions with the system, and that of other learners like them, to deliver an effective and engaging learning experience. Technology isn’t the answer, but it is part of the best solution.

Thanks for reading. I encourage you to watch the whole talk as it has many more gems than I’ve noted here, and several laugh out loud moments.

QCommons opens to all content areas

Today I rolled out a big update to QCommons, implementing  support for many more content areas than Chemistry.

The key new feature is groups. Each group has its own set of content, its own forums, and its own classification terms. (Because “rational” doesn’t mean the same in Economics as it does in Algebra.) Each group can also have its own permissions system and administrators to support more private uses such as school district curriculum committees. If you would like to host a new group, please contact

To keep a handle on the growth of the site, I’ve switched registrations to requiring administrator approval. Please register and sign up for the mailing list. You’ll go into a queue that I’ll process regularly to allow more users.

Stay tuned to this blog for more!

Introducing QCommons: Chemistry

After a long summer of programming and design, the first output of the QCommons research project is ready for the world.  I will blog more about QCommons in general, but today I’d like to tell you about QCommons: Chemistry, made in partnership with the Chemistry Education Digital Library.

In a nutshell, QCommons is a platform for assessment resources that promotes sharing and collaborative improvement.  It is open source and designed to support many different topic domains.  To track developments with QCommons please fill out the subscription form currently on the front page of the site.

QCommons: Chemistry is the first site built on the platform.  In developing sharing sites, there is a bootstrapping issue: people are unlikely to contribute to a site that doesn’t already offer something.  This makes starting out tough, but once it gets rolling the network effects help it grow faster and faster.  (Wikipedia being a prime example.)  Fortunately, the ChemEd DL and the Journal of Chemical Education have donated hundreds and hundreds of quality assessment items to get the ball rolling.  Anyone can come and browse through the items for ones useful in their formative or summative assessments.  All the items are from the JCE QBank and are used in General Chemistry courses at major universities. These were previously available only to vetted teachers, but now the questions (without the answers) are available to everyone.  (At this point, to see the answers you will still need to prove you are a teacher.  I welcome feedback on how to improve this.)

This is just the beginning.  There are many topic areas, new features and usability improvements to come.  In the spirit of Google’s long “beta” cycle, I’m releasing the site as it is to let people start using it and give feedback. It is ready to use and relatively bug free.  As I add new features I intend to keep it that way.  Just bear in mind that it’s a work in progress and any suggestions you have could strongly influence how it develops.

So again, all comments, criticisms and suggestions are welcome.  You can post them here on this blog or in the forums on the site.

OER Copyright Survey

Educators enjoy certain greater freedoms under copyright law’s Fair Use Doctrine, and such the culture of exchange in education is pretty lax.  In a workshop recently in which I discussed Qcommons (more on that later this week), the teachers said that if they were reticent to upload their materials because they’d lost track of what was original and what they’d copied from copyrighted sources.  They knew they had the right as teachers to use them in their courses, but did not know whether they could publish them online, and thus were effectively silenced by the intimidating complexity of copyright law.

Creative Commons has set out to make copyright and licensing terms easier to understand and use by everyone.  CCLearn is trying to do the same for education, which I argue is an even greater task.  Today they put up a survey to get a sense of how people understand the copyright rules and how they work with them.  Please take 5-10 minutes to fill it out and help advance access to and sharing of educational resources.  It closes August 31 so go ahead and do it now.  🙂

From the survey page,

“Because most content remains “all-rights-reserved” under the traditional rules of copyright, it is often the case that the creators and producers of OER must confront questions as to when and if it is permissible to use content created by others when it is not offered under an open license. For example, an OER creator may want to incorporate a clip from a film into a lesson about film techniques, or an animated video illustrating a biological process into a lesson about that process. However, if the film clip or animation is protected by “all-rights-reserved” copyright, then the OER creator may be unsure how to proceed, or may wish to rely on some exception to copyright law that may apply under such circumstances.

It is our goal to develop a deeper awareness of the degree to which OER practitioners and users grapple with copyright law issues, and whether those issues pose barriers to the creation, dissemination, and reuse of OER. We hope that this initial survey will form the basis of a larger international study led by ccLearn.”

Usability testing gets easier and easier

Your product is useless if no one can use it.  But how do you know how usable it is?  You can try it yourself, but you are not the user. (“You are not the user” is practically a mantra here at HCII.)  One set of methods, usability inspection, provides a framework to systematically evaluate a system yourself or with the help of others versed in the methods.  But really, you want to know how actual users experience the system.

Usability testing is a systematic way to see how real users get along with your design.  There is much literature around usability testing, which you may want to read.  Or you can just dive in using some of the easy tools that are available these days.

If you have access to users and can put them in front of your computer, there is software that can record both the screen and a webcam at the same time so that you can go back later to watch and listen to the user while they’re using your software. The best is Silverback, which is pretty slick but only for Mac.  (For Windows, keep looking.)   Because it’s recorded, you can skip over boring stuff and replay interesting stuff over and over.  You don’t even need to be there to record it, if other people are helping you.

Then there’s remote testing. If you’re testing a web application, your users don’t even have to leave their computer.  Today I came across these remote testing services and they’re pretty great:

userfly is impressive and also the cheapest.  You add a Javascript string to your page and the everything the user does on the page is recorded.  Then they play it back for you, showing the mouse move around, clicks and keypresses.  It has some difficulties updating with results of AJAX calls, but they’re working on that.  They offer 10 recordings per month free, so it’s worth trying. is like a mix of Silverback and userfly.  For $29, they provide the user for you and record a video of their screen, with an audio track of them thinking aloud using your website.  Apparently the user also provides a written summary of the problems they found.  I haven’t tried it since it costs money, but I wonder who the users are.  I expect they’re experienced evaluators, which helps in some ways, but can also be detrimental.  If you’re targeting a special population, then you probably want to find the users yourself.

Chalkmark has a very specific purpose, seeing where people click on an image when given an instruction. E.g. you upload an image of your web site and tell them “click on the link to update your settings”.  Wherever they click gets recorded, along with everyone else’s clicks on the task, to create a heat map on the image.  I guess it’s useful if you’re carefully testing out the layout of a site across a large population, but usually a small sample suffices and you would get better data with the other tools.

Remember, you are not the user and your assessments of the usability of what you made are likely way off.  Test with real users.  These tools make it easy.

Conference deadline extension statistics

Anyone who publishes in conferences knows the pain of making sacrifices to meet a deadline, only to find at the last moment that it has been extended.  Year after year of these extensions and people come to expect them, making them all the more inevitable.  This reminds me of people who set their clocks fast to be late less, only to adjust themselves again to be late and have to move their clocks ever earlier.  At what point do conference deadlines lose their credibility?

I’d be interested to know how many people believe a conference’s deadline, as an effect of how often that conference extends its deadlines.  I imagine that younger researcher are more credulous, but that it’s simply because they lack the knowledge of all the extensions.

Are there data available on deadline extensions? If not, why not collect it? Each time a deadline is extended, there is surely at least one person who is frustrated.  What if there were a place they could go and vent their frustration by adding the occurrence to some listing/database?  Well here is the place:

To add your data, or vent depending on how you see it, just fill out this form:

I hope this helps. I just added AERA, which had been due today.

Plone, Drupal, Moodle and ATutor

I’m overdue for an update here on what I’ve been working on.  My current project is a web application for collaboratively sharing, critiquing and improving question items for homeworks and quizzes. Basically a wiki question bank, but more socially-oriented.

In my last post, I was working on Plone and the ECQuiz module of eduComponents.  After a few months I abandoned Plone and never looked back. Plone’s going through a big transition right now and it’s hard to be a newcomer to its scene.  I wish Plone and eduComponents developers well.

I switched my platform to Drupal and its Quiz module. Drupal has an amazing community.  It’s hard to measure a community, but a handy data point is that the Drupal group on Facebook has 3500 members, compared to Plone’s 500.  This Quiz module has an active forum, an IRC channel, and a longish history.

While the Quiz community is strong, its design is lacking for my purposes.  It began 3 years ago as a simple module and has been pulled and contorted over the years to suit different needs.  This is arguably the best way for an open source module to evolve.  Thanks to big contributions from Matt Butcher, the module got some big improvements in Quiz 3.0.  For example, there is now an object-oriented type system for question types so new ones can be added more easily.  (Unfortunately, Drupal data schemas don’t have inheritance like PHP objects do so data properties of the base class have to be included in each subclass.  Unless someone wants to hack around that.)  And now Sivaji is making yet more improvements for Quiz 4.x, as part of his winning GSoC proposal.  Quiz 4.0 will be a polished set of improvements at the end of this summer.

I’ve been exploring the potential for Quiz to take a leap forward by drawing in code from PHP-based learning management systems.  I began with Moodle and hammered its import/export code into Quiz to allow it to handle many more formats.  I was happy having materialized the possibility for re-use in open source, but overall pretty turned off by Moodle’s spaghetti codebase.  Maybe if you’re a longtime Moodle developer it all seems clean and clear, but that wasn’t my experience.  So I’ve kept looking.

ATutor looks promising.  It’s a leaner codebase and looks so far to be a clean design.  It also has much better support for standards, which is important for my question bank in order to interoperate with other systems.  After skimming the source code I realized I would need the db schemas to wrap my head around it so I installed the whole thing on my laptop.  Wow, that was easy.  I just moved the folder within my MAMP htdocs, navigated to it in my browser, and the rest was clicking through web forms.  (There was one step where I had to make a directory manually, but the directions were explicit enough for anybody.)  Thanks ATutor developers, and Happy Canada Day.

I may post again with an assessment.  Please reply in the comments if you’d like to hear this (and why would also help).

Plone tips

I’ve chosen Plone for a web application I’m working to develop, a collaborative site for educational assessment materials.  I have years of experience with PHP/LAMP and Java, but am pretty new to Python and very new to Plone.

As such, I’ve been having to figure out a lot along the way and I thought I should share some of what I’ve learned to save others the trouble.  Plone 3.2 introduced some big improvements to the Plone build system recently that make it much easier to get started with and that aren’t in any books yet.  (My copy of Professional Plone Development by Martin Aspeli covers Plone 3, but not 3.2. Plone’s doc on how to upgrade to 3.2 only helped a little.  )

The PSU WebLion wiki has been very helpful.  They have a great description of what a Buildout is, among other helpful info.  That’s where I found the this blog entry on building out with Plone 3.2.x, which now packages Plone as Python eggs.

Here’s a minimal buildout procedure:

$ mkdir plone3.2_buildout
$ cd plone3.2_buildout
$ wget*checkout*/zc.buildout/trunk/bootstrap/
$ cat > buildout.cfg
extends =
versions = versions
find-links =
parts = 

recipe = plone.recipe.zope2install
url = ${versions:zope2-url}
fake-zope-eggs = true

recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
eggs = 
$ python2.4
$ bin/buildout


For any other Plone products you use, check to see if they’re in Pypi.  If so, you can just add them to the list of eggs in the instance.  If your favorite product isn’t in egg form, consider nudging the developers. Plone recommends all Products become eggs and it’s easy to do. I know hardly anything about Plone and was able to convert the ECQuiz product to an egg.  If you want that egg you can just add "Products.ECQuiz" under "Plone" in your instance:eggs list. (Tip for converters:  if you get "unbound prefix" from configure.zcml you probably need this line,

    xmlns:genericsetup="" )

I’m also making edits to the ECQuiz code so it’s a little more complicated, but only barely.  I keep the code in src/ and I added the buildout.eggtractor extension to my buildout.cfg which scans the src directory for product eggs and automatically adds the appropriate values to the buildout.  Add it like so,

  extends =

  versions = versions


  extensions =

If you get, "Error: Missing option: buildout:develop" you also need a "develop = " option to [buildout].  For more on buildout, try the Buildout Quick Reference card.

Now let’s say you want to check your new site into Subversion.  (This tip I got from Martin Aspeli’s slideshow on Extending and Customizing Plone 3.)  You don’t want to check your whole directory in because much of that is built out automatically.  Other developers don’t need it and likely won’t even work on their system because much of what is built out contains hard-coded paths to other files and executables.

So before checking all the files within your buildout, update the directory’s svn:ignore property to omit the stuff that doesn’t need to go into version control.  In your buildout directory run,

$ svn propset svn:ignore '
> eggs
> develop-eggs
> bin
> var
> parts
> .installed.cfg' .  # note period for current dir
$ svn add products buildout.cfg src README.txt
$ svn commit

Now someone else can check out that code and run "python" to generate the bin directory and then "bin/buildout’ to build out all the eggs and code.

If you’re building more than one Plone site, you can save disk space and unnecessary downloads by configuring your Buildout to use a shared directory.

$ mkdir -p ~/.buildout/downloads
$ mkdir -p ~/.buildout/eggs
$ cat > ~/.buildout/default.cfg
  eggs-directory = ~/.buildout/eggs

  download-cache = ~/.buildout/downloads

That’s all I got for now.  I hope that’s helpful.

Pasteur’s and Edison’s Quadrants

If you’ve been to an plenary session or keynote on education research chances are you’ve heard of Pasteur’s Quadrant.  It’s the idea that basic science (e.g., Bohr) and applied science (e.g., Edison) can be brought together to have significant impact on society, as exemplified in Pasteur’s prodigious contributions. Which quadrant should education research target? Pasteur’s Quadrant was articulated (and I believe originated) in the book of the same name by Donald Stokes in 1997. Stokes was a professor of politics and public affairs and argued for use-inspired basic research as an important target of public funds. My interest is more in research than public policy and I haven’t read the book. If you’re interested, but the publisher’s summary gives a good summary of the public policy thesis and the transitions in government research funding that it spoke to.


Education research, to my eyes, is still finding and defining itself. After Dewey, the greatest advances were by the psychologists. That would be “basic” research in education. While we learned a lot about how people learn, little of it made it into classrooms. Other education research was unscientific, with anthropological or post-modernistic perspectives. There wasn’t much education science and the little that there was was quite basic. The No Child Left Behind act shook this all up. (Quick poll: do you pronounce NCLB as en-see-el-bee or nickleby? I met someone recently who calls it nickleby.) Part of the act formed the Institute for Educational Science within the Dept of Ed. “Educational Science,” now what does that mean? The first director of IES, Grover (Russ) Whitehurst, in a speech describing the mission of IES, cited Pasteur’s Quadrant. But he went further and called for IES to focus on Edison’s Quadrant. It’s worth a read. (I think this is fair use…)

One way of making this distinction is in the terms introduced in the infrequently read but oft cited 1997 book by Stokes, called Pasteur’s Quadrant – Basic Science and Technological Innovation. Stokes described three categories of research based on two binary dimensions: first, a quest for fundamental understanding, and second, a consideration of use. The work of the theoretical physicist, Niels Bohr, exemplifies the quadrant in which researchers search for fundamental knowledge, with little concern for application. The research of Louis Pasteur, whose studies of bacteriology were carried out at the behest of the French wine industry, characterizes the work of scientists who, like Bohr, search for fundamental knowledge, but unlike Bohr, select their questions and methods based on potential relevance to real world problems. The work of Thomas Edison, whose practical inventions define the 20th century, exemplifies the work of scientists whose stock and trade is problem solution. They cannibalize whatever basic and craft knowledge is available, and conduct fundamental research when necessary, with choices of action and investment driven by the goal of solving the problem at hand as quickly and efficiently as possible.

Considerations of Use
Low High
Quest for Fundamental Understanding Yes Pure Basic Research (Bohr) Use-Inspired Basic Research (Pasteur)
No Pure Applied Research(Edison)

Each of the scientific quadrants identified by Stokes is important to the common good. Those who argue for the value of basic research have no trouble finding examples of work inspired only by intellectual curiosity that turned out to be extremely practical. Bohrs’ work on quantum physics is a case in point. Without in any way diminishing the value of basic research, whether use-inspired or not, I want to argue for the importance of activities in Edison’s quadrant, particularly for topics in which there is a large distance between what the world needs and what realistically can be expected to flow from basic research, and for topics in which problem solutions are richly multivariate and contextual. Education is such an area: a field in which there is a gulf between the bench and the trench, and in which the trench is complicated by many players, settings, and circumstances. Choose what you consider to be the most exciting developments from basic research in Bohrs’ or Pasteur’s quadrants that are relevant to education. I’ll pick developments in cognitive neuroscience. Paint the rosiest scenario you dare for basic scientific progress in the topic you’ve chosen over the next 15 years. Then ask yourself what would need to be done to translate those imagined findings into applications that would have wide and powerful effects on education outcomes. I don’t know about you, but I’m not optimistic that the results of basic research, even if the findings are powerful, will flow directly and naturally into education. Goodness! Education hasn’t even incorporated into instruction what we know from basic research about the effects of massed versus distributed practice – and I learned about that in a psychology course I took in 1962. Yes, the world needs basic research in disciplines related to education, such as economics, psychology, and management. But education won’t be transformed by applications of research until someone engineers systems and approaches and packages that work in the settings in which they will be deployed. For my example of massed versus distributed practice, we need curricula that administrators will select and that teachers will follow that distributes and sequences content appropriately. Likewise, for other existing knowledge or new breakthroughs, we need effective delivery systems. The model that Edison provides of an invention factory that moves from inspiration through lab research to trials of effectiveness to promotion and finally to distribution and product support is particularly applicable to education. In summary, the Institute’s statutory mission, as well as the conceptual model I’ve just outlined, points the Institute toward applied research, Edison’s quadrant. I’ve labeled this chart, “Edison’s quadrant, mostly,” because I understand that it is important to nurture the development of basic knowledge related to education, particularly in areas in which other science agencies and major foundation’s aren’t involved. Thus, when resources permit, the Institute will support work that examines underlying process and mechanisms, and work that is initiated by the field. For instance, the President’s budget request for the Institute for fiscal year 04 includes a healthy amount of money for a field-initiated competition. In addition, many of our new funding programs that are squarely focused on application, such as our program in preschool curriculum evaluation, provide for grantees to carry out parallel research that examines underlying processes.

What do you think? I would love to hear your thoughts in the comments.  Here’s another quote that might inspire you, Nikola Tesla speaking of Edison:

If Edison had to find a needle in a haystack, he would proceed with the diligence of a bee to examine straw after straw until he found the object of his search… I was almost a sorry witness of his doings, knowing that just a little theory and calculation would have saved him 90 per cent of the labor…

Incidentally, searching online, I see Pasteur’s Quadrant cited in ICT, chemistry and even geology. (Evidently, there is such a thing as applied geology.) Try those links if you want to know more about Pasteur’s Quadrant generally. And by the suggestion of my colleague Andrea Forte during the OLI Symposium 2008, there’s a Wikipedia article. Please improve upon it as I only made it last night. 😉