PewePro 2

Student Activity Monitoring

A method for both document's and system's fragment identification through implicit user feedback.

Inputs: Pictures of user from ordinary webcam, read wear, other implicit interest indicators
Outputs: Identified fragments

Addressed Problems

Identification of important or interesting parts of content is useful feature of adaptive learning system as it can be used for various tasks ranging from document summarization and recommendation to adaptive guide to system itself. Common approach is to evaluate explicit user feedback, however user (student) has to devote his time to providing the feedback, which is distracting him. Also well-known biases exist, e.g. U- or J-shaped rating curve.


With implicit feedback we let the user work uninterrupted and track his actions - clicks, time spent on different portions of document (virtually wearing the document out - read wear), etc. However, time based indicators are sensitive to user presence and his other activities, i.e. user can leave the computer with document open. We therefore employ gaze tracking using common webcam, which allows detection of times when user is not present or is performing different activities and also serves as another interest indicator.

We proposed a method for important fragment identification using various implicit interest indicators. For this purpose we divided interest indicators into several groups and assigned weight to each group and to each indicator independently. Examples of indicators in each groups are the following:

  1. Untargeted - mouse cursor movements, ...
  2. Passively targeted - content being displayed (read wear), gaze,
  3. Actively targeted - text selection, text copying, domain-specific actions (annotating, etc.), ...
  4. Document-related - document prints, number of visits, ...

Where by targeting we mean whether user does this action mostly when he is working with related fragment (e.g. cursor movement alone is untargeted, but text selection is targeted) and by passively/actively targeting we mean whether user is just looking at the content or he is actively working with it (e.g. copying it).

To collect designed indicators in web environment, several approaches exists - from custom implementation of own web browser, which is unpractical, to tracking client-side code implemented in ALEF. More universal solutions include tracking proxy and tracking browser extension. We implemented the latter approach with desktop application for gaze tracking integrating with browser extension via local client-server model communication.

Also fragments were divided into two main groups of those related to a document and those outside of document (elements of application itself), with more subgroups in document-related group - e.g. when the document is learning object of type Exercise, subfragments are its parts Definition, Hint and Solution, et cetera.

For such fragments we compute Attention Index, where we sum up weighted values of each indicator in each group separately, sum up weighted groups and normalize by length of fragment in question. With fragments composed of subfragments, normalization is done by entire length of fragment only after summing up its subfragments' computed without normalization. Values of indicators are seconds in time based indicators (e.g. gaze time), count of specified user actions in action based indicators (e.g. mouse clicks), length (e.g. selected text length) or combinations (e.g. count of selections and their lengths).

We proposed several scenarios for use of this method:

  1. Interesting fragments - identified interesting fragments can be visualized for use as quick reference when repeating the already learned content or when reading the content for the first time. Such fragments can be either highlighted directly, summarization of original document can be performed by selecting only such fragments or even adaptive scrollbar can move the document according to importance of displayed content.
  2. Adaptive guide to learning application - information about users' work with web-system can be used to aid him with the learning, i.e. by providing him with clues about other functions of the system. If user is not using recommendation for example, personalized message about this feature can be presented - we can differentiate:
    1. Whether user has already noticed this feature, but ignored it - we can ask him why he is not using it and get feedback whether he does not want to use it in general or we picked unsuitable items for him.
    2. Whether user did not notice this feature with gaze at all - then we can modify overall design to make this feature more prominent if it is general problem or just pinpoint it for users like him.
  3. Augmented communication - instant messaging communication of students visiting the same learning object can be augmented with information about actions of their peers. As students should more easily contact others when they see that they are stuck on the same problem, cooperation is encouraged this way.


  1. Labaj, M.: Web-Based Learning Support based on Implicit Feedback. In: Information Sciences and Technologies Bulletin of the ACM Slovakia. ACM Slovakia Chapter. Vol. 3, No. 2., 2011, pp. 76-78.