11-767 On-Device Machine Learning

On-Device Machine Learning is a project-based course covering how to build, train, and deploy models that can run on low-power devices (e.g. smart phones, refrigerators, and mobile robots). The course will cover advances topics on distillation, quantization, weight imprinting, power calculation and more. Every week we will discuss a new research paper and area in this space one day, and have a lab working-group the second. Specifically, students will be provided with low-power compute hardware (e.g. SBCs and inference accelerators) in addition to sensors (e.g. microphones, cameras, and robotics) for their course project. The project will involve three components for building low-power multimodal models:
        (1) inference
        (2) performing training/updates for interactive ML, and
        (3) maximizing power.
The more that can be performed on device, the more privacy preserving and mobile the solution is.

For each stage of the course project, the final model produced will have an mAh "budget" equivalent to one full charge of a smart phone battery (~4 Ah or 2hrs on Jetson Nano, 7hrs on RPi, or 26hrs on a RPi Zero W).

Time & Place: 10:10am - 11:30am Tu/Th in GHC 4307
Course questions and discussion: Slack -- Registered students will be added
GitHub template for assignments: https://github.com/strubell/11-767
Assignments will be submitted and graded using Canvas

Example Industry Motivation "... if the coffee maker with voice recognition was in use for four years, the speech recognition cost for chewing on data back in the Mr Coffee datacenter would wipe out the entire revenue stream from that coffee maker, but that same function, if implemented on a device specifically tuned for this very precise job, could be done for under $1 and would not affect the purchase price significantly. " -- Source

Instructors

Yonatan Bisk

Instructor

ybisk@cs.cmu.edu

Emma Strubell

Instructor

strubell@cmu.edu

Ankit Shah

aps1@andrew.cmu.edu

Slack and Course Communication

All course communication will happen via slack including slides and discussions. Private correspondence with instructors should happen via email.

Slack

#general-questions: For questions about lectures, the course, or help from others on class projects
#group-N: Each group should come up with a name and create their own channel (invite TA and instructors). Use the same name for your GitHub fork and pin the link to the channel. Please also invite us to the GitHub. Example: group-RoboFun
#hardware: Hardware related questions should be asked broadly for anyone to help with.
#paper-discussion: Questions/thoughts about weekly readings should happen publicly here.
Private Messages: If there is a question you would like to address to the instructors, please email 11-767-instructors@cs.cmu.edu. Also, please check #general-questions first and post there when possible.

Policies and Grading

The course is split half on paper discussion and half projects.

Papers	Project/Lab
– Participation	20%	– Lab Reports	45%
– Paper Presentations	15%	– Final Report & Presentation	20%

Participation:
Participation in Class or Slack #hardware/#paper-discussion (20%)
Participation is evaluated as "actively asking/answering questions based on the lectures, readings, and/or assisting other teams with project issues". Concretely, this means that every novel question or helpful answer provided in Slack will count for 1%, up to a total of 20% of your grade.

Submission Policies:

Submit a link to the markdown/PDF in your GitHub repo on Canvas.
Lab reports and final projects will be written up and submitted as a group. Each member of the group should submit on Canvas for each assignment, but they should all link to the same writeup.
All deadlines are 5pm EST (determined by Canvas submission time).
Late days: Every team has a budget of 5 late days to be used throughout the semester. They will be automatically calculated based on submit time, after which 2% absolute per day is removed from max grade.

COVID Details:
In the event a student tests positive for COVID-19, they will be invited to attend discussion virtually and will be expected to participate as usual. This includes participation points for raising their hands with questions/answers and submission of lab-notebooks. Note, that students who attend class while exhibiting symptoms will be told to leave and join virtually for the protection of all others present.

Accommodations for Students with Disabilities:
If you have a disability and have an accommodations letter from the Disability Resources office, we encourage you to discuss your accommodations and needs with the instructors as early in the semester as possible. We will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, we encourage you to contact them at access@andrew.cmu.edu.

Note to students

Take care of yourself! As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, feeling down, difficulty concentrating and/or lack of motivation. This is normal, and all of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of a healthy life is learning how to ask for help. Asking for support sooner rather than later is almost always helpful. CMU services are available free to students, and treatment does work. You can learn more about confidential mental health services available on campus through Counseling and Psychological Services (CaPS). Support is always available (24/7) at: 412-268-2922.

Take care of your classmates and instructors! In this class, every individual will and must be treated with respect. The ways we are diverse are many and are fundamental to building and maintaining an equitable and an inclusive campus community. These include but are not limited to: race, color, national origin, caste, sex, disability (visible or invisible), age, sexual orientation, gender identity, religion, creed, ancestry, belief, veteran status, or genetic information.

Research shows that greater diversity across individuals leads to greater creativity in the group. We at CMU work to promote diversity, equity and inclusion not only because it is necessary for excellence and innovation, but because it is just. Therefore, while we are imperfect, we ask you all to fully commit to work, both inside and outside of our classrooms to increase our commitment to build and sustain a campus community that embraces these core values. It is the responsibility of each of us to create a safer and more inclusive environment. Incidents of bias or discrimination, whether intentional or unintentional in their occurrence, contribute to creating an unwelcoming environment for individuals and groups at the university. If you experience or observe unfair or hostile treatment on the basis of identity, we encourage you to speak out for justice and offer support in the moment and/or share your experience using the following resources:

Center for Student Diversity and Inclusion: csdi@andrew.cmu.edu, (412) 268-2150
Report-It online anonymous reporting platform (user name: tartans; password: plaid); (877) 700-7050

Projects, Hardware, and Resources

The course will be primarily centered on a few multimodal tasks/platforms to facilitate cross-team collaboration and technical assistance. If your team wants to use custom hardware or sensors not listed here -- that's fine, but please reach out so we can discuss it and put think through the implications. Every team will also be provided with one of the following Single Board Computers (SBCs)

Example Projects

Input	Output	Task
Speech	Text	Open-Domain QA
Images	Text	Object Detection or ASL Finger Spelling
Images	Robot Arm	Learning from Demonstration
Speech + Images	Robot Car	Vision-Language Navigation

Single Board Computers

SBC	RAM	Notes
Raspberry Pi 4	8GB	2Amp draw on moderately powerful processor
Jetson Nano	2GB, 4GB	128-core NVIDIA Maxwell CUDA cores

FAQ

Can we use other platforms? Yes! But let's talk about the pros and cons together.
What about custom sensors and hardware? Same answer :)
Will the course pay for a Coral/Pi Zero/...? Yes, but let's chat first.

Resources

Course Schedule

A basic course schedule is presented below. Papers will be chosen jointly by the instructors and the student presenting.

Discussion	Lab
Aug 31: Course structure & Background	Sept 2: Hardware landscape and trade-offs
Sept 7: Intro to practical applications Natural Language Processing with Small Feed-Forward Networks Recognizing People in Photos Through Private On-Device Machine Learning	Sept 9: Hardware setup & run inference Lab Report 0 practice
Sept 14: Compression I: Distillation & Pruning The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	Sept 16: Project expectations and brainstorming Lab Report 1 5%
Sept 21 Project Proposal Workshopping (Lab) Project Proposal Due	Sept 23: Neural Architecture Search (Discussion) EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks HAT: Hardware-Aware Transformers for Efficient Natural Language Processing Once-for-All: Train One Network and Specialize it for Efficient Deployment
Sept 28: Benchmarking Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures Showing Your Work Doesn’t Always Work Evaluating the Energy Efficiency of Deep Convolutional Neural Networks on CPUs and GPUs	Sept 30: Benchmarking Lab Report 2 5%
Oct 5: Compression II: Quantization Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 I-BERT: Integer-only BERT Quantization Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers	Oct 7: Quantization Lab Report 3 5%
Oct 12: Multimodal Fusion Early Fusion for Goal Directed Robotic Vision VisualBERT: A Simple and Performant Baseline for Vision and Language	Oct 14: No class
Oct 19: Architecture specific tricks I: CNNs XOR-Net: An Efficient Computation Pipeline for Binary Neural Network Inference on Edge Devices MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications	Oct 21: Implement baselines and related work Lab Report 4 5%
Oct 26: Architecture specific tricks II: Transformers A Study of Non-autoregressive Model for Sequence Generation Do Transformer Modifications Transfer Across Implementations and Applications? Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing	Oct 28: Group Work Lab Report 5 5%
Nov 2: Guest lecture: Shinji Watanabe Streaming End-to-end Speech Recognition For Mobile Devices	Nov 4: Group Work Demo by Alan Black Lab Report 6 5%
Nov 9: Accelerating Training Pre-Training Transformers as Energy-Based Cloze Models Prefix-Tuning: Optimizing Continuous Prompts for Generation MLPerf Training Benchmark	Nov 11: Group Work Lab Report 7 5%
Nov 16: Paper Presentations I Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level A Primer in BERTology: What We Know About How BERT Works The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain TBD	Nov 18: Group Work Lab Report 8 5%
Nov 23: Paper Presentations II Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT TBD	Nov 25: No class
Nov 30: Carbon and the Future The Hardware Lottery	Dec 2: Carbon calculation and final report/presentation prep Lab Report 9 5%
Dec ??: Final Presentation during finals slot	Dec 10: Final Report due

11-767: On-Device Machine Learning

Fall 2021

Instructors