Learning with Social-Emotional Robots


Orpheum Children’s Science Museum in Champaign-Urbana has a great resource upstairs: a closet packed full of LEGO mindstorms parts. Dozens of motors and sensors, a classroom set of Mindstorms CPUs and laptops to program them, and bins & bins full of parts.

When I started volunteering as an assistant, I was really pleased with the diversity of activities Greg Fabry stuffed into the robotics camp sessions: gum drop toothpick engineering challenges, blindfold-your-friends and ‘programming’ their movement through an obstacle course, and a good old egg drop. There was certainly a lot of learning going on, from social+teamwork to mechanics intuition (building strong things out of weak materials is a wonderful foundation for engineering).

However, I always thought it was a shame the kids didn’t get more hands on time with the expensive robotics tools that were unique to that space. The class met twice a month for less than 2 hours per session, so the actual hands-on experience with robotics was very modest. So, the next year when I was asked to lead the class, I decided I would try my own off-the-cuff teaching with as much robot building and programming as possible.

The usual LEGO bot. Photo credit: nxtprograms.com

Over the course of 2 or 3 sessions with the classic curriculum of building a simple rover and programming it to navigate obstacles, I made a few observations of how a classroom of mixed skill-sets interacts with these robots.

The kids that have experience reading LEGO build instructions (which involves lots of counting, unless you’ve already mastered estimating the lengths at a glance) get ahead while newbies make mistakes that necessitate taking the robot apart (very discouraging for a newbie). So you’ve already got a split classroom where some kids maybe get the feeling that they’re not in the right place.

And it’s the same story with programming: with 2–4 kids per laptop, it’s hard to get around the fact that the kid who’s done it before is going to take the mouse from a newbie struggling to solve the introductory problems, unless you specifically tell them not too, in which case maybe they feel frustrated and slowed down. You can get past that OK by setting the expectation at the beginning of class that people who know how to do something should instruct their less experienced teammate by pointing at the screen, not taking away the mouse, but I still never really liked the dichotomy, and it just got worse as the programming challenges quickly progressed from pre-programmed route to line-following robot. The previous curriculum had each 2 hour session introducing a more difficult programming challenge, but that quickly gets too difficult for kids who have never looked at a program before.

With all of that in mind, I set out to come up with a project that all the kids could start with a fresh slate — a build with programming challenges that none of them had encountered yet. That’s where the robot face pictured above comes in.

https://www.youtube.com/watch?v=X8g4dJPTvKo

I decided being able to turn a motor on to make a mouth open and close + being able to twist some eyebrows up and down would allow for a lot of fun with really basic programs. I take a cue from Rachel Hellenga’s teaching style in thinking that it’s much better for everyone to use the basics in a lot of different ways instead of moving onto very specific uses of more-difficult-to-absorb knowledge.

The best part about building the robot face (A Lego CAD model to help you build your own is here RobotFaceV2.lxf, the software to view it is available from Lego here) is that it allows modular construction where groups can decide how to delegate the parts: advanced kids can pick the challenge of assembling the more complex linkages of the mouth and beginners can have fun picking out different parts to make eyeballs (they certainly don’t have to look like the example robot — so they get to be a little creative here.) If I remember right, some components were even challenging for the adult helpers, and having adults and kids work together on a problem that’s difficult for both of them is my favorite 😉

So our first lesson was simply getting things moving, turning motors on, changing the speed of the motor, learning that you can go 1 revolution at one speed, and the next revolution at another speed, and build patterns of these motor blocks to create patterns in the robot’s movement. Without much prompting there was much puppeting and kids coming up with things to say to accompany the robots miming (typical “I AM A ROBOT” stuff, but still lots of fun.) Also it’s lots of fun to make the robot’s eyebrows wiggle.

Before I had an idea of what the next activity would be, programmer-extraordinaire Caden had set up a couple of buttons to control the robot’s movements live and used his remote control face to tell a knock knock joke. After that it was pretty obvious what we could spend a few weeks on. The next session, each group was prompted to keep their laptops closed, no programming yet! One by one, tell your best knock knock joke to your group. Then decide amongst yourselves who’s got the funniest knock knock joke and in a few minutes we’ll ask each group to tell their joke to the class.

This was my favorite robotics class ever because it got started with everyone laughing at each others jokes. Kids get some ‘public speaking’ practice with each other and have to make a decision as a group, which is a good thing to get practice with, too.

Once each group settled on a joke, they were instructed to program their robots to tell the joke as puppets. They were encouraged to try different strategies, whatever it took to get their robots to talk to one another. This facilitated the different skill levels by allowing different solutions to the same problem. Some students got to try out simply lining up ‘move’ blocks with different speeds, some students got to try using timers and events, some students knew how to use conditionals to add those remote-control buttons, but everyone got to try what they knew how to do and share the result with their teammembers: because even the simplest program solved the problem of making the robot talk, there was no sense of being left behind or being slowed down. Students got to see how other teams were working and asked what their programs did (and getting one student to explain their program out loud to another is really great to see.)

When it came to testing the programs, reciting knock knock jokes along to your miming robot face was a lot more fun to fail than having your line-following robot fail to follow a line. For some reason, when trying to line up the pace of your speech with the pace of the robot’s mouth movements, it’s really funny for the robot to stop moving in the middle of a word, or to keep moving after you’re done talking — I guess because the robot flubbed a joke? Something about it had the kids laughing and I had never been in a programming class where debugging what went wrong was that funny.

Trying out lots of different ways to do the same thing also meant everyone was ready to have a crack at presenting their project at the end of the day — it was pretty much impossible to fall behind, you could just tell your joke using whatever program you had working. Getting a camera and a clapperboard (“Action! *CLAP*”) is always fun for kids, and we recorded a few takes of each groups knock knock joke.



The second iteration of the robot face involves a motor for each eyebrow and a piston mechanism that results in different eyebrow movements depending on which direction the motor is turned (the force of the motor will push or pull on the end of the eyebrow and change what direction it’s pointed.)

I finished designing this face pretty much the day before I wanted to use it, so unfortunately instead of creating build instructions I just built 4 copies. But that meant that the session after knock knock jokes could jump right into giving the robots emotions.

Before programming the robot to express emotions, we had to learn what different emotions looked like. So the activity went like this:

  • Sit everyone at a long table so each person has someone sitting directly across from them. It doesn’t matter if the kids are paired with adults or other kids.
  • Give everyone on one side of the table paper and markers.
  • Instruct everyone on the opposite side of the table to pick an emotion and express it on their face.
  • Instruct the drawing side to draw the expression with the fewest lines possible: just get the shape of the eyebrows and mouth first.
  • Ask the drawing side to guess what emotion the face-maker is trying to express. Just happy, sad, surprised, angry, that kind of thing.
  • Have everyone check their answers with the person across from them and then switch sides.

Don’t make a big deal out of it if someone can’t pick a face to make or can’t make a guess. The important thing for the kids is to spend some time focusing on an expression and practice identifying what it means. This is intuitive to some people, but lots of people have to learn what emotions facial expressions correspond to. At the end of the activity, students will have compiled a list of drawings and corresponding emotions.

The programming activity now becomes moving each eyebrow motor into position to match a drawing. This can also be a lesson in how 60 degrees compares to 180 degrees and so on, because the eyebrows only need to be moved a few degrees to change expression.

This expressive robot face now allows for a storytelling activity: think about what emotions you can get your robot to express, and write a story for the robot to tell (from his/her own perspective, i.e., come up with an identity and backstory for your robot) that would incorporate an emotion. Extra points if the robot’s emotion changes in the story and has its eyebrows move accordingly.

This allowed some of the students who were timid about building mechanisms or writing programs to express their creativity: the quietest kids had some really surprising stories for their robots to tell, and it was really exciting to see each kid decide what role they wanted to take in the project: changing the robot physically (some kids spent their time building props out of LEGO to go with their story — or adding red lights to the face to be laser cannons on their bad guy robot), writing code to control the face, or writing a script and drawing backdrops for the story.

It was a pretty long process to write and produce a story told by robots, so we didn’t get to the point of recording finished projects, but I have high hopes for this format of robotics education in the future.

Hacking Music at the People’s Music School

Our attempt at Montessori-style music composition with blocks

With a team of new friends, I got to build a musical instrument that combines LEGOs and Arduino. An ultrasonic distance sensor hangs under a gantry and plays different notes according to the LEGOs underneath it. Our hurried first build was fragile, but pulled off a successful live demo in front of the audience.

(We called it FORTissimo since you could build LEGO forts that make loud noises.)

Thanks to Nora’s suggestion to join the facebook group Hackathon Hackers, I found out about an event I couldn’t pass up: a music/technology hackathon put on by a the People’s Music School (@PeoplesMusicSch ). Only trouble was I was living 300 miles away at a cabin in the woods of Southern IL. Luckily I wasn’t too far from an Amtrak station, and I scheduled a roundtrip on the City of New Orleans inside of the 24 hours I had off work (it seemed that way at the time).


After a few hours of spotty sleep on the train, and a drowsy cab ride to Merchandise Mart, I can’t tell you how refreshing it was to walk off the elevator into a sunlit atrium filled with the sounds of a string quartet. Throughout the day a few students of the music school would set up in one room or another to give a soundtrack to the brainstorming and building. It was awesome.

This was the first hackathon I’d participated in. The way teams were divvied up was a bit awkward: after opening remarks, the audience was asked if anyone had an idea for a team yet — perhaps they were expecting more small groups to show up ready to go, but I think most of the people came as individuals. So instead a few people introduced themselves and said what comes to their mind when they think about tech / music / education. I introduced myself and said I liked to use Arduino to teach music and coding.

In the coffeecake fueled team-forming mingling that came afterward, I was approached by a couple different people who had ideas for software to help people learn music, and I ended up sitting down with a couple of women who had kids studying music — it was a lot of fun to think about what makes practicing difficult for anyone (I had dropped out of music school a few years before partly because I couldn’t quite stomach the 4 hours a day in the practice booth that my professors recommended).

We identified that the repetition of material coupled with the lack of feedback makes sitting in a practice room a pretty dull experience. So we brainstormed ways of automating feedback — perhaps a program could take a recorded example of a teacher playing a passage, and compare a student trying to copy it. I still think a computer could quickly point out wrong notes and wrong timing so that you don’t practice the same passage incorrectly over and over, but we agreed that replacing a music teacher with an app was pretty pie-in-the-sky.

People’s movement from one team to another was pretty fluid, which was great — it meant ideas got around. Having people work around each other openly, brainstorming on white boards makes it pretty easy to join a conversation, and not long after, join a team. After bouncing ideas off a few different people during the morning, I came across a table scattered with LEGOs. Alex and Ania had brought their hack-kit with them and already had some beeps going on while constructing lego towers. Of course I sat down to ask them what they had going on. Alex had seen an Instuctables for an Arduino Distance Sensor with Buzzer and LED before coming to the hackathon and brought his Arduino, distance sensor, and piezo buzzer. After catching me up, he complained that his main concern is that the buzzer wasn’t a loud enough output. I showed him that it was really easy to control the sound from a laptop via Arduino, but he insisted that he doesn’t want the product to be tethered to a computer to work. Luckily (well, I was going to a music-tech hackathon) I had speakers and transistors in my laptop bag and built two simple one-transistor amplifiers to output some simple square wave tones. We rubber-banded the mini breadboard to the top of their gantry. Happy with the improvement, they let me join their team and commence a couple hours of programming and debugging to get each note of a major scale to be triggered by a particular height of LEGOs.

Note the kid in the background who looks jealous that we got to play with LEGO.

The live demo went great, playing a melody up and down a major scale (cleverly avoiding the bug we never worked out when going from the highest tower back down to nothing). Among the rest of the presentations, ours was the only one that was more than theoretical (though there were some web mockups that were pretty impressive for 6 hours of design time). I’d still like to explore software that compares a student practicing to some standard and tells the student what they can do to improve. I’d also like to actually build FORTissimo as a kit. It really would be pretty cheap and versatile. But for now it was just a fun weekend.

Code is at https://github.com/jazzyjackson/FORTissimo/