On-Device Machine Learning is a project-based course covering how to build, train, and deploy models that can run on low-power devices (e.g. smart phones, refrigerators, and mobile robots). The course will cover advances topics on distillation, quantization, weight imprinting, power calculation and more. Every week we will discuss a new research paper and area in this space one day, and have a lab working-group the second. Specifically, students will be provided with low-power compute hardware (e.g. SBCs and inference accelerators) in addition to sensors (e.g. microphones, cameras, and robotics) for their course project. The project will involve three components for building low-power multimodal models:
        (1) inference
        (2) performing training/updates for interactive ML, and
        (3) maximizing power.
The more that can be performed on device, the more privacy preserving and mobile the solution is.
For each stage of the course project, the final model produced will have an mAh "budget" equivalent to one full charge of a smart phone battery (~4 Ah or 2hrs on Jetson Nano, 7hrs on RPi, or 26hrs on a RPi Zero W).

  • Time & Place: 9:30am - 10:50am Tu/Th in DH 1212
  • Assignments will be submitted and graded using Canvas
Example Industry Motivation "... if the coffee maker with voice recognition was in use for four years, the speech recognition cost for chewing on data back in the Mr Coffee datacenter would wipe out the entire revenue stream from that coffee maker, but that same function, if implemented on a device specifically tuned for this very precise job, could be done for under $1 and would not affect the purchase price significantly. " -- Source


Yonatan Bisk
Yonatan Bisk



Emma Strubell
Emma Strubell



If there is a question you would like to address to the instructors, please email 11-767-instructors@cs.cmu.edu.

TA: Clara Na, OH: Tuesday 1-2pm in GHC 5417
TA: Jared Fernandez, OH: Thursday 3-4pm in Wean 3110


Submission Policies:
COVID Details:
In the event a student tests positive for COVID-19, they will be invited to attend discussion virtually and will be expected to participate as usual. This includes participation points for raising their hands with questions/answers and submission of lab-notebooks. Note, that students who attend class while exhibiting symptoms will be told to leave and join virtually for the protection of all others present.

Accommodations for Students with Disabilities:
If you have a disability and have an accommodations letter from the Disability Resources office, we encourage you to discuss your accommodations and needs with the instructors as early in the semester as possible. We will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, we encourage you to contact them at access@andrew.cmu.edu.

Note to students

Take care of yourself! As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, feeling down, difficulty concentrating and/or lack of motivation. This is normal, and all of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of a healthy life is learning how to ask for help. Asking for support sooner rather than later is almost always helpful. CMU services are available free to students, and treatment does work. You can learn more about confidential mental health services available on campus through Counseling and Psychological Services (CaPS). Support is always available (24/7) at: 412-268-2922.

Take care of your classmates and instructors! In this class, every individual will and must be treated with respect. The ways we are diverse are many and are fundamental to building and maintaining an equitable and an inclusive campus community. These include but are not limited to: race, color, national origin, caste, sex, disability (visible or invisible), age, sexual orientation, gender identity, religion, creed, ancestry, belief, veteran status, or genetic information.

Research shows that greater diversity across individuals leads to greater creativity in the group. We at CMU work to promote diversity, equity and inclusion not only because it is necessary for excellence and innovation, but because it is just. Therefore, while we are imperfect, we ask you all to fully commit to work, both inside and outside of our classrooms to increase our commitment to build and sustain a campus community that embraces these core values. It is the responsibility of each of us to create a safer and more inclusive environment. Incidents of bias or discrimination, whether intentional or unintentional in their occurrence, contribute to creating an unwelcoming environment for individuals and groups at the university. If you experience or observe unfair or hostile treatment on the basis of identity, we encourage you to speak out for justice and offer support in the moment and/or share your experience using the following resources:

Bring your own Data/Hardware/Projects for Lab

The first lab will be done individually on your laptops to ensure everyone is comfortable with the basic libraries. Afterwards, teams will form and propose what data or task they are interested in using as their benchmark for the course. As part of this, we will provide several options for Single Board Computers (SBCs) (see below), but many teams will have a preferred platform they would rather use (e.g. their own robot). This is completely fine, but we will verify during the early benchmarking labs that experiments can be run in those settings.

Example Projects
Input Output Task
Speech Text Real-time Machine Translation
Images Text Object Detection or ASL Finger Spelling
Images Robot Arm Learning from Demonstration
Speech + Images Robot Car Vision-Language Navigation
Side view of three-wheel robot car LCD Screen and Camera for ASL
Single Board Computers
Raspberry Pi 4 8GB 2Amp draw on moderately powerful processor
Jetson Nano 2GB, 4GB 128-core NVIDIA Maxwell CUDA cores

Course Schedule

A basic course schedule is presented below. Papers will be chosen jointly by the instructors and the student presenting.
Aug 29: Course structure & Background
Aug 31: Lab 1: PyTorch, MLP, Basic Benchmarking (individual assignment)
Sept 5: Practical Problems, Theory vs Practice
Sept 7: Machine Learning and Optimization
Sept 12: Efficiency Benchmarking
  • Sep 13: Lecture Highlight
Sept 14: Project Group Speed dating
  • Sept 15: Lab 1 Due
Sept 19: Computer Vision, Edge Hardware, and Robotics Sept 21 NLP
Sept 26: Compression I: Quantization and Pruning
  • Sep 27: Lecture Highlight
Sept 28: Lab 2 + Project Hours: Benchmarking Your Data, Model, Hardware
Oct 3: Compression II: Pruning (cont) and Distillation
  • Oct 4: Lecture Highlight
Oct 5: Neural Architecture Search
  • Oct 6: Lecture Highlight
  • Oct 6: Project Proposals Due
Oct 10: Architecture-specific tricks I: CNNs
  • Oct 11: Lecture Highlight
Oct 12: Lab 3: Quantization
  • Oct 13: Lab 2 Due
Oct 17: Fall Break Oct 19: Fall Break
Oct 24: Project Sharing Oct 26: Project Sharing
  • Oct 27: Project Sharing Write-up Due
Oct 31: Architecture-specific tricks II: Transformers
  • Nov 1: Lecture Highlight
Nov 2: Project Feedback (in class!)
  • Nov 3: Lab 3 Due
Nov 7: Election Day Nov 9: Lab 4: Pruning
Nov 14: Efficient Training (or don't)
  • Nov 15: Lecture Highlight
Nov 16: Parameter Efficient Fine-Tuning
  • Nov 17: Lecture Highlight
Nov 21: Project Hours (zoom option available)
  • Scoping and status discussions (optional)
  • Nov 21: Lab 4 Due
Nov 23: No class: Thanksgiving
Nov 28: Environmental Impacts
  • Nov 29: Lecture Highlight
Nov 30: Lab 5: New Hardware + Benchmarking Energy
Dec 5: Final Presentations (and sharing) Dec 7: --
  • Dec 7: Hardware return (except late days)
  • Dec 8: Lab 5 Due