E-Chalk is software we developed in order to enhance the classroom experience.  Our guiding philosophy was to provide the teacher with a tool to make lectures more dynamic and interesting. At the same time, the lectures can be stored and transmitted live over the Internet. E-Chalk transforms a large touch-sensitive screen into a smart teaching tool. The instructor writes on the screen using a special stylus: the color and width of the line strokes can be changed so the screen emulates a classical chalkboard. However, the lecturer can do much more with E-Chalk:  images from bookmarks can be pasted to the screen, a query to a remote Web service can be sent, an algebraic server can be asked to plot a function, Java applets with animations can be pasted to the board.  A copy of the lecture (audio, board, and video) is stored on a server and the lecture can be also transmitted live over the Internet. A remote viewer needs only a conventional Java-capable browser in order to view or replay a lecture. A PDF transcript of the lecture is available from an automatic repository.

The E-Chalk architecture is based on the metaphor of the classical chalkboard, enhanced by intelligent assistants running in the background, and providing various services. One assistant is responsible for interpreting the handwritten input of the user. Another is a mathematical formula recognizer that processes handwritten queries for the algebraic server. A circuit simulator recognizes sketches of digital circuits and runs a simulation. An algorithm simulator accepts sketches of graphs as input data and runs graph algorithms, animating them on the screen. Additional intelligent assistants can be interfaced by E-Chalk savvy users by just interfacing their applications to E-Chalk using the API we provide.

The E-Chalk project was funded by the Deutsche Forschungsgemeinschaft (DFG) and the Klaus-Tschira-Foundation. Several students work on the project under my supervision. Dr. Lars Knipping (now TU Berlin) and Dr. Gerald Friedland (now in Berkeley) obtained their PhD degrees with this project.

Software Architecture for a Smart Teaching Board

In our project the board components of E-Chalk and an interface for third-party applications are being developed. Intelligent software agents look over the shoulder of the lecturer and handle various types of input. In 2007, we ported the system to MPEG4 so that E-Chalk lectures can now be played on computers, cellular telephones, and IPods.

The E-Chalk server software currently consists of three main and several minor components. The three main components are the audio, the video, and the board server. The minor components are the database manager and the PDF converter. The E-Chalk client system consists of a set of independent receiver applets that synchronize themselves by communicating through our Media Applet Synchronization Interface (MASI).

E-Chalk has basically two ways of operating: online for live transmissions and offline for archived transmissions. During live transmissions the audio stream, video stream, and board events are recorded from their devices, compressed, and sent. They are then received, decompressed and replayed in real time. While this is being done, everything that is sent out is also stored in files. As a result, the E-Chalk client system has two modes. In live mode, each client connects to its corresponding server through a socket connection. In on-demand mode, clients use an http connection to receive the files and no E-Chalk server is needed.

We are working on interfacing more third-party programs. Mathematica, Maple, and other programs can already process input coming from the board. Algorithmic animations of handwritten input are now possible. We also have a circuit simulator that can evaluate logical circuits drawn with AND, OR, and NOT gates. All these applications rely on the E-Chalk API we have developed. The API allows a program to handle handwritten input from E-Chalk and to request strokes to be drawn on the board. We call this program “intelligent assistants”.

 Intelligent Audio and Video Processing for E-Chalk

In our project we developed software to increase the quality of the audio channel and to merge the board and video channels in the future.

The quality of the audio channel suffers during lecture hall recordings for several reasons (noise, echo, reverberance, etc.). It can be improved by using special hardware and a mixer. But in our case, we want to provide the user a way of producing high-quality recordings purely by adapting the software. Our software is an expert system for audio recordings that extracts a fingerprint of the noise from the hardware, as well as a fingerprint of the recording situation. The fingerprint is used to eliminate artifacts and to modulate the recording level. The results can compete with expensive hardware.

In the second part of the project we want to track and segment the lecturer from the video channel. The segmented lecturer can then be transmitted together with the board data and can be superimposed on the board. The effect is that the board image is reproduced in high quality (as board strokes) and the lecturer’s facial expression is still visible. The viewer can even dim the lecturer in order to see through him.

In this project we are extending state of the art algorithms for lecturer recognition and tracking. We need to be able to cut the lecturer image in real-time. Reconstruction at the receiving end must also be done in real time. A first version of the lecturer segmenter has been written: it can segment the lecturer at 10 frames per second. A version for tracking objects in screenplays is being written by an open source group affiliated to GIMP.

Recognition of Handwritten Mathematical Formulas

In our project we are developing a mathematical formula recognizer for E-Chalk. The idea is to give the lecturer a tool for evaluating formulas on-the-fly during a class. The formula recognizer can also be used to transform handwritten formulas into LeTeX code.

When a formula is drawn free-hand on the board, the recognizer first identifies and groups individual strokes. Once the strokes belonging to a character have been selected, the next step is to simplify their shape, which is done by eliminating intermediate points in sequences along lines. Only some points are kept  in curves, i.e. those which can best approximate the curve piecewise within a certain tolerance factor.

 When a character has been reduced to simplified strokes, represented by just a few points each, we encode each stroke as a vector of features.  Once each stroke has been encoded as a vector, we train classifiers for each character (taken from a training set produced by several people). The classifiers are neural networks or support vector machines. There is a classifier for each different number of strokes (since the number of encoding vectors is different). The classifiers recognize each letter and provide their ASCII code.

Finally the two-dimensional layout of the formula must be recognized. We do this using the minimum spanning tree of the symbols, modified by some appropriate heuristics. The formula recognizer is currently in use in E-Chalk and JMath, the formula recognition shell, can be used to produce LeTeX code for handwritten formulas.

Examples of mathematical formula recognition. The LaTeX code is shown on the respective lower right subframes, the typeset formula on the respective upper right subframes.

This project was pursued by Ernesto Tapia (postdoc) under my supervision and with funding from DFG (Deutsche Forschungsgemeinschaft)

Algorithmic Animation for E-Chalk

Algorithmic animation is the art of producing a visual image of the operation of a certain algorithm. Several algorithmic animation systems have been written, but none of them has ever handled handwritten input. In this project we will write the pseudocode for an algorithm on the electronic chalkboard. The computer will then recognize the handwriting of the user, will compile the program and semi-automatically produce the required animation. In the figure below, for example, the user has written a program. Encircling the variables A and i tells the system that we want to run the program seeing a visual representation of A and i. The array A is shown as a collection of bars. The variable i is shown as an arrow, because it is an index to the array A.

It is already possible to run algorithms in E-Chalk, for which the input is entered as a sketch. Dijkstra’s shortest path algorithm can be programmed and made to run on a graph entered by the user by hand. The figure below shows an example.

Teaching Mathematics with a Data Wall

We have been teaching mathematics and natural sciences at Freie Universität Berlin using a large data wall composed of four computer screens. Fig. 1 shows the four screens arranged in a row. The LCD screens are bright enough to be operated in the classroom without dimming the lights. The contrast of the screens provides a much better visual experience than a chalkboard. Students sitting in the last row can still see the diagrams and formulas clearly.

The lecturer writes on a contact sensitive tablet, as shown in Fig.2.  This allows the lecturer to talk directly to the class keeping eye contact.

The four screens and the tablet are connected to a desktop computer retrofitted with a special extension board for controlling the five screens (the tablet is the fifth screen). As shown in the pictures above, the lecturer can use different colors and line widths. Images can be pasted on the screens.  The images come from a pre-assigned folder in the hard disk or just from the clipboard (allowing cut-and-paste from a browser, for example). The screens do not have to be “erased”, when the lecturer runs out of space the image just scrolls up. The lecturer can scroll up or down during the lecture to show previous material again.

The complete lecture is stored and is available right after the class in two formats: a) As a PDF file of the complete lecture, b) In a special format which allows “replaying” the lecture at home. The user at home just starts the player in the browser and the class is shown like a video on the user’s screen, including the audio signal captured with an ambient microphone.

The program running on the five screens has been called IntelliChalk (for intelligent chalk). It is, first, a painting program which provides the lecturer the tools to draw and write with high quality. But it is also a program which manages images for pasting, as well as scans of handwritten notes. Usually mathematics lecturers prepare notes for their class on paper. These notes can be digitized using a digital camera mounted on the desk (Fig. 3). The individual class fragments are separated by a horizontal line (see Fig. 4). The software digitizes the ink on the notes and separates the notes in different files, which we call “macros”. The macros are available for the lecturer in the classroom computer. The lecturer can then just paste definitions, or the statement of a theorem, and just write the proof or development he or she is interested to explain, saving time during class.

Fig. 4: The photograph of the lecturer’s notes and the digitized macros

Our vision for the future is having a data wall composed of several screens and with almost invisible seams, so as to have a large two-dimensional space where a class can be thought. Fig. 5, below (prepared using Photoshop), shows how we imagine that large screen for the immersive classroom.

The data wall can be used to show PowerPoint slides, videos or any kind of visual material. LCD data walls will substitute large projection systems in universities and research institutes within the next ten years.

The IntelliChalk project aims at developing the software for such new data walls including the following features:

• Slides presentations are possible
• Interactive lecturing is possible. Science classes can be held using a contact sensitive screen mounted on a podium.
• Handwriting recognition can be used to start secondary applications such as simulators, algebraic servers, or an image search over the Internet.
• Videos can be pasted to the board.
• The students in the classroom can annotate their own local copy of the class material using their own tablets. The student’s annotations constitute an additional information layer.
• The lecture is available over the Internet as a file for printing, or as a file for replaying the lecture (screen contents and audio)
• Video of the lecturer is available as an additional channel.
• The handwritten notes of the lecturer can be digitized in a few seconds before the lecture starts (mounting a digital camera as digitizer in the podium).
• And much more.

IntelliChalk could be used in conference mode for delivering classes through the Internet, live or differed.

We are also experimenting with 360-degree videos of the lectures. This video material will be superposed on the digital media. Since we capture the complete lecture room, omnidirectionally, we are investigating the social editing (that is the collaborative editing) of the videos. The image below shows some of the videos we have captured with a six cameras omnidirectional video system and the partial views which can be generated by the software.

Several persons are collaborating in this project: Dr. Ernesto Tapia (postdoc) and three more students. The project is funded by FU Berlin.