english only
EPFL > STI > IMT > LASA > Teaching > Advanced Machine Learning
LASA MENU
Home
Research
Publications
Teaching
People
Arriving to LASA
Videos / Photos
Press & News

GitHub Repository
Older Source Codes
The Blog
Intranet

Prospective Students
TEACHING INDEX
Teaching
Student Projects
Former Projects
Robot Contests

Advanced Machine Learning




Overview and objective

The aim of machine learning is to extract knowledge from data. The algorithm may be informed by incorporating prior knowledge of the task at hand. The amount of information varies from fully supervised to unsupervised or semi-supervised learning. This course will present some of the core advanced methods in the field for structure discovery, classification and non-linear regression. This is an advanced class in Machine Learning; hence, students are expected to have some background in the field. The class will be accompanied by practical session on computer with the following software package:




People Lecturer: Aude Billard, Teaching Assistants: Nadia Figueroa, Ilaria Lauzana, Brice Platerrier
Grading Scheme 50% of the Grade will be based on personal work done during the semester (this entails either doing a computer based mini-project or writing a literature survey, see section Practicals below). The remaining 50% of the grade will be based on a 40 minutes oral exam (20 minutes preparation, 20 minutes oral defense). The oral will examine the student on the material viewed during the course.
Prerequisites

This course is intended as an advanced class in ML for MSc and PhD students and hence focuses on advanced topics of ML as well as on critical reading of the state of the art in ML. Student who want to take this course are expected to have taken some introductory ML course, such as the applied ML course given at the master during the winter semester.

The course assumes general strong knowledge in Linear Algebra, Probability and Statistics.

Time and Location

Every other week, we alternate between:

  • 2 hours Lectures and 2 hours exercises / practicals. These are held on Fridays (10:15-12am) and (3:15-5pm)

Check detailed schedule below to know which one takes place when!

Lectures + exercises take place in room:
(Morning Sessions) CM104.
(Afternoon Sessions) CM010

Afternoon Practical Sessions will take place in Room ME.D2.2419

Practice sesions will be done in Room ME D2 2419, PCs are provided. However, students can bring their own laptop and have related software packages installed.

Software The course encompasses computer-based practicals centred on the use of ML_toolbox. An open-source software which allows to visualize and test most of the major algorithms seen in class in MATLAB.


Practicals and Mini-Project

Practicals
The practicals will cover different topics (Kernel Methods, Classification and Regression). In addition, students will conduct either a mini-project or a literature survey during the course of the semester.

Mini-Project/Lit.Survey
The mini-project will entail implementing an algorithm of choice and one or more variants, evaluating the algorithm performance and sensibility to parameter choices. The literature survey will be done on a topic chosen among a list provided in class. The student will read and analyze a choice of recent articles on the topic and provide a written summary. The choice will be announced to the registered students during the semester.

A page dedicated to the mini-project can be found here: Spring 2017 Mini-Project Webpage

Important Dates:

Sign up for lit. survey/mini-project must be done by March 10 2017.
Reports for lit. survey/mini-project must be handed out by May 19 2017.
Oral presentations will take place on May 26 2017 .


Lecture Schedule and Timetable


Date Related Documentation Course Topics Exercises/TP Template Solutions Exercises
Week 1
Feb 24th
(10:15am-12pm)


Concepts

(Classification, Regression, Pattern Recognition,...)


Feb 24th
(3:15pm-5pm)
Probability
Expectation Maximisation


Recap Statistics
(Probability Distribution, Likelihood, E-M, etc.)



Week 2
March 3rd
(10:15am-12pm)

Kernels


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernels


Code to draw isolines (MATLAB)

March 3rd
(3:15pm-5pm)


Introduction to Mini-Projects
Mini-project + Lit. Review Topics
(Room CM010)



Week 3
March 10
(10:15am-12pm)
kPCA


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernel PCA


 

KPCA Exercise Solutions

March 10
(3:15pm-5pm)


Practical Session 1
Dimensionality Reduction: PCA + Kernel-PCA
(Room ME D2 2419)
TP1-PCA+kPCA


MATLAB Code TP1


TP1-PCA+kPCA
Solution


Week 4
March 17th
(10:15am-12pm)
kCCA


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernel CCA



KCCA Exercise Solutions


March 17th
(3:15pm-5pm)
Kernel K-means


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernel K-means



Week 5
March 24th
(10:15am-12pm)
Tutorial on Spectral Clustering
Spectral Methods recap


Spectral methods
(Structure Discovery, Dim. Reduction)

Spectral Clustering


Spectral Clustering Exercise Solutions

March 24th
(3:15pm-5pm)


Practical Session 2
Manifold Learning + Spectral Clustering
(Room ME D2 2419)

TP2-Manifold+Clustering


MATLAB Code TP2


TP2-Manifold+Clustering
Solution


Week 6

March 31th
(10:15am-12pm)


Classification

Support Vector Machine (SVM)


March 31th
(3:15pm-5pm)

Sparse Bayesian Learning and the RVM (Tipping 2011)
RVM
SV Clustering


Classification

Support Vector Machine and extensions



Week 7
April 7th
(10:15am-12pm)


Classification

Bagging, Boosting + Ransac


Bagging, Boosting and Ransac Exercise Solutions

April 7th
(3:15pm-5pm)


Practical Session 3
Classification: SVM + Boosting
(Room ME D2 2419)

TP3-Classification


MATLAB Code TP3

TP3-Classification
Solution

Week 8
April 14th

+++ Easter Break +++
Friday April 14 -- Sunday April 23



Week 9
April 21st

+++ Easter Break +++
Friday April 14 -- Sunday April 23



Week 10
April 28th
(10:15am-12pm)


Regression

Non-linear regression techniques: SVR + RVR


 

Supplementary Exercises

Solutions

April 28th
(3:15pm-5pm)


Supplementary Lecture - Imbalanced Datasets

Imbalanced Datasets


Practical Session 4a
Regression: SVR + RVR
(Room ME D2 2419)

TP4a-Regression


MATLAB Code TP4a

TP4a-Regression
Solution

Week 11

May 5th
(10:15am-12pm)

GPR
GPR Model Selection


Regression

Non-linear regression techniques: GPR, Gradient Boosting


Supplementary Exercises

Solutions

May 5th
(3:15pm-5pm)


Mini-Project Practical Session
Work on mini-project / lit. survey



Week 12
May 12th
(10:15am-12pm)
V and Q recursions


Supplementary Lecture - Reinforcement Learning

Introduction to Reinforcement Learning


May 12th
(3:15pm-5pm)


Mini-Project Practical Session
Work on mini-project / lit. survey




Week 13
May 19th
(10:15am-12pm)


Supplementary Lecture - Time-Series Analysis

HMM extensions


 

May 19th
(3:15pm-5pm)


Practical Session 4b
Regression: BLR + GPR + GradBoost
(Room ME D2 2419)
TP4b-Regression


MATLAB Code TP4b


TP4b-Regression
Solution


Week 14
May 26th
(10:15am-12pm)
 


Student Oral Presentations of Projects / Surveys
Please attend to your colleagues' presentations!



May 26th
(3:15pm-5pm)


Student Oral Presentations of Projects / Surveys
Please attend to your colleagues' presentations!



Week 15
June 2nd
(10:15am-12pm)

Overview of Class

June 2nd
(3:15pm-5pm)

Q&A Session


3 - 4 July
Room CM 0 10

Oral Exam

Scheduled via doodle sign-up (available on moodle).





Online Resources in Machine Learning

Machine Learning.org

On-line repository of papers from NIPS conference

Pascal: Network of Excellence on Pattern Recognition, Statistical and Computational Learning (summer schools and workshops)

ML List: "Archives of the Machine Learning List"

ML Repository: "This is a repository of databases, domain theories and data generators that are used by the machine learning community for the empirical analysis of machine learning algorithms."

MLnet: Machine Learning network online information service. "This site is dedicated to the field of machine learning, knowledge discovery, case-based reasoning, knowledge acquisition, and data mining."

KDnet: "The KDNet (= Knowledge Discovery Network of Excellence) is an open Network of participants from science, industry and the public sector. The major purpose of this international project is to integrate real-life business problems into research discussions and to collaborate in shaping the future of Knowledge Discovery and Data Mining."

Clever Methods for Overfitting


Recommended Textbooks

Selected chapters from some of the book below are made available to the class, see column "related documentation" above.

General Introduction to Machine Learning

Machine Learning: A probabilistic perspective by Kevin P. Murphy, MIT Press

Introduction to machine learning by A. Smola and S.V.N. Vishwanathan, on-line version.

"Bayesian Reasoning and Machine Learning" by D. Barber, Cambridge University Press

 

Kernel Methods: PCA, SVM:

 - "Kernel Methods for Pattern Analysis" by John Shawe-Taylor, Nello Cristianini
   Publisher: Cambridge University Press (June 28, 2004)
  
 - "Pattern Recognition and Machine Learning" by Christopher M. Bishop
   Publisher: Springer; 1 edition (October 1, 2007)

- "Learning with Kernels", B. Scholkopf and A. Smola, MIT Press 2002
  

Statistical Learning Methods:

 - "Pattern Classification" by Richard O. Duda, Peter E. Hart, David G. Stork
   Publisher: Wiley-Interscience; 2 Sub edition (October 2000)
  
 - "Introduction to Statistical Learning Theory" by Olivier Bousquet, Stephane Boucheron, and Gabor Lugosi
   http://www.kyb.mpg.de/publications/pdfs/pdf2819.pdf

- "Information Theory, Inference and Learning Algorithms", David J.C Mackay, Cambridge University Press, 2003.

 

Neural Networks:

- "Spiking Neuron Models” by W. Gerstner and W. M. Kistler
Publisher: Cambridge University Press, Aug. 2002

- "Hebbian Learning and Negative Feedback Networks (Advanced Information and Knowledge Processing) by C. Fyfe.
Publisher: Springer.

- "Independent Component Analysis", A. Hyvarinen, J. Karhunen and E. Oja, Wiley Inter-Sciences. 2001.

- "Self-Organizing Maps", Teuvo Kohonen, Springer Series in Information Sciences, 30, Springer. 2001.

  - "Introduction to Neural Networks: A Comprehensive Foundation" (2nd Edition) by S. Haykins.

 

Reinforcement Learning

- "Reinforcement Learning: An Introduction",. R. Sutton & A. Barto, A Bradford Book. MIT Press, 1998.

- "Reinforcement Learning: A Survey", Leslie Pack Kaelbling & Michael L. Littman and Andrew W. Moore, Journal of Artificial Intelligence Research, Volume 4, 1996., 1996



Last update: 06/04/17