Project
Electro TA
A hackathon project my roommate and I built for Gemini's Live Agent Challenge. The task was to build an AI agent with multimodal inputs and outputs. We designed a live Electrical Engineering Teaching Agent that can help debug hardware in real time.
Getting a usable image to the agent
We used an ESP32 devkit microcontroller, alligator clips for reading voltage signals, and an Arducam to send pictures to the agent. To make a conversation feel seamless, you can't have large pauses between speaking. An early hardship was latency with the Arducam: we wanted to send pictures of the student's breadboard circuit to Gemini, and not being able to send a photo quickly would prolong every response.
We had to learn about two communication protocols the Arducam uses simultaneously: I2C and SPI. I2C handles the internal register of the camera, while SPI handles downloading the actual JPEG bytes. Finding the balance between increasing image resolution and still being able to transmit it was very difficult. A lot of testing went into finding the perfect ratio of resolution and baud rate, as well as focusing the camera. We also added code to improve the saturation of the image, which was crucial for getting a comprehensible picture to the agent.
A mini "multimeter" on the ESP32
We also wanted to send real time information about the circuit to Gemini. Using the ADC pins on the ESP32, we built our own mini "multimeter." It works by taking the absolute voltage at a node (not the voltage drop) and scaling it down through a small voltage-divider circuit to feed it into the ESP32.
We learned this the hard way when our first ESP32 got fried from reading too high a voltage (5V at the time). The ESP32 ADC pins can only handle 0–3.3V, so to read a 5V signal we had to scale the voltage down and then multiply by that same scaling factor in the program to recover the correct reading. That scaling factor is about 2.04, just basic circuit algebra, and you can see it in the ESP32 firmware.
Safeguards
To make the system more robust we added both hardware and software safeguards. On the hardware side, a Zener diode protects the board: if the voltage exceeds 3.3V, the Zener enters breakdown and dumps the excess into ground, protecting the circuit. On the firmware side, since ADC readings on the ESP32 can be very noisy, we used oversampling (rapidly taking 20 voltage readings and averaging them) to get a stable number to send to the agent.
The software
Two scripts run with the microcontroller. The first is written in C++ (in PlatformIO) and tells the microcontroller how to capture pictures and read voltage. The second is a Python script that saves the image and the voltage at a given time, then sends that information to the live agent.