Yonatan Bisk

Instructor

ybisk@cs.cmu.edu

Emma Strubell

Instructor

strubell@cmu.edu

Policies and Grading

• Submit a link/PDF to Canvas.
• Lab reports are individual while projects will be written up and submitted as a group.
• All deadlines are midnight EST (determined by Canvas submission time).
• Late days: Every team has a budget of 5 late days to be used throughout the semester. They will be automatically calculated based on submit time, after which 2% absolute per day is removed from max grade.

Note to students

• Center for Student Diversity and Inclusion: csdi@andrew.cmu.edu, (412) 268-2150
• Report-It online anonymous reporting platform (user name: tartans; password: plaid); (877) 700-7050

Projects, Hardware, and Resources

Example Projects

Input	Output	Task
Speech	Text	Real-time Machine Translation
Images	Text	Object Detection or ASL Finger Spelling
Images	Robot Arm	Learning from Demonstration
Speech + Images	Robot Car	Vision-Language Navigation

Single Board Computers

1. Can we use other platforms? Yes! But let's talk about the pros and cons together.
2. What about custom sensors and hardware? Same answer :)
3. Will the course pay for a Coral/Pi Zero/...? Yes, but let's chat first.

Resources

• TinyML by Warden and Situnayake 2019
• Getting Started with AI on Jetson Nano
• PyTorch Mobile
• ONNX Mobile
• Example DexArm Rotary Code

Course Schedule

Example Readings

• Green AI
• Energy and Policy Considerations for Deep Learning in NLP
• Early Fusion for Goal Directed Robotic Vision
• Knowledge Transfer for Efficient On-device False Trigger Mitigation
• Distilling the Knowledge in a Neural Network
• DistilBERT
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable NAS
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• High-Performance Large-Scale Image Recognition Without Normalization
• Show Your Work: Improved Reporting of Experimental Results
• Showing Your Work Doesn’t Always Work
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• Bayesian Bits: Unifying Quantization and Pruning
• Scalable Methods for 8-bit Training of Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Mapping Navigation Instructions to Cont. Control Actions with Position-Visitation Pred.
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Efficient softmax approximation for GPUs
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Do Transformer Modifications Transfer Across Implementations and Applications?
• A Primer in BERTology: What we know about how BERT works
• Efficient Transformers: A Survey
• Improving Low Compute Language Modeling with In-Domain Embedding Initialisation
• Consistent Accelerated Inference via Confident Adaptive Transformers
• PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
• Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
• Are Sixteen Heads Really Better Than One?
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• I-BERT: Integer-only BERT Quantization
• Natural Language Processing with Small Feed-Forward Networks
• Recognizing People in Photos Through Private On-Device Machine Learning
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• The Hardware Lottery
• Streaming End-to-end Speech Recognition For Mobile Devices
• Pre-Training Transformers as Energy-Based Cloze Models
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• MLPerf Training Benchmark
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• MiniVLM: A Smaller and Faster Vision-Language Model
• Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning
• Dataset Distillation
• XOR-Net: An Efficient Computation Pipeline for Binary Neural Network Inference on Edge Devices
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures
• Showing Your Work Doesn’t Always Work
• Evaluating the Energy Efficiency of Deep Convolutional Neural Networks on CPUs and GPUs
• A Study of Non-autoregressive Model for Sequence Generation
• Do Transformer Modifications Transfer Across Implementations and Applications?
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing