By Louis DiPietro, Cornell Ann S. Bowers College of Computer and Information Science
A Cornell team has developed an interface that allows users to handwrite and sketch within computer code – a challenge to traditional coding that typically relies on typing.
The pen-based interface, called Notate, allows users of computer-based, digital notebooks – such as B. Jupyter notebooks, which are web-based and interactive – to open artboards and handwrite diagrams within lines of conventional, digitized computer code.
Based on a deep learning model, the interface bridges handwritten and textual programming contexts: the notation in the handwritten diagram can refer to text code and vice versa. For example, Notate recognizes handwritten programming symbols like “n” and then maps them to their typewritten equivalents. In one case study, users drew quantum circuit diagrams in Jupyter notebook code cells.
The tool was described in “Notational Programming for Notebook Environments: A Case Study with Quantum Circuits” presented at the ACM Symposium on User Interface Software and Technology held October 29-November 2 in Bend, Oregon. The paper, whose lead author is Ian Arawjo, a PhD student in the field of information science, received an honorable mention at the conference.
“A system like this would be great for data science, especially for sketching charts and diagrams that then work together with text code,” Arawjo said. “Our work shows that the current programming infrastructure is actually holding us back. People are ready for this kind of feature, but code-entering interfaces developers need to take note and support images and graphical interfaces within the code.”
Arawjo said the work shows a new way forward by introducing artificial intelligence-based, pen-based coding at a time when drawing tablets are becoming more widely used.
“Tools like Notate are important because they give us new ways of thinking about what programming is and how different tools and representation practices can change that perspective,” said Tapan Parikh, associate professor of information science at Cornell Tech and a paper co-author .
Additional co-authors include: Anthony DeArmas ’22; Michael Roberts, graduate student in computer science; and Shrutarshi Basu, Ph.D. ’18, currently Visiting Lecturer in Computer Science at Middlebury College.
Louis DiPietro is a writer at Cornell Ann S. Bowers College of Computing and Information Science.
This story originally appeared in the Cornell Chronicle.
