english only
EPFL > STI > IMT > LASA > Teaching > Advanced Machine Learning
LASA MENU
Home
Research
Publications
Teaching
People
Arriving to LASA
Videos / Photos
Press & News

GitHub Repository
Older Source Codes
The Blog
Intranet

Prospective Students
TEACHING INDEX
Teaching
Student Projects
Former Projects
Robot Contests

Advanced Machine Learning




Overview and objective

The aim of machine learning is to extract knowledge from data. The algorithm may be informed by incorporating prior knowledge of the task at hand. The amount of information varies from fully supervised to unsupervised or semi-supervised learning. This course will present some of the core advanced methods in the field for structure discovery, classification and non-linear regression. This is an advanced class in Machine Learning; hence, students are expected to have some background in the field. The class will be accompanied by practical session on computer with the following software package:




People Lecturer: Aude Billard, Teaching Assistants: Nadia Figueroa, Ilaria Lauzana, Brice Platerrier
Grading Scheme 50% of the Grade will be based on personal work done during the semester (this entails either doing a computer based mini-project or writing a literature survey, see section Practicals below). The remaining 50% of the grade will be based on a 40 minutes oral exam (20 minutes preparation, 20 minutes oral defense). The oral will examine the student on the material viewed during the course.
Prerequisites

This course is intended as an advanced class in ML for MSc and PhD students and hence focuses on advanced topics of ML as well as on critical reading of the state of the art in ML. Student who want to take this course are expected to have taken some introductory ML course, such as the applied ML course given at the master during the winter semester.

The course assumes general strong knowledge in Linear Algebra, Probability and Statistics.

Time and Location

Every other week, we alternate between:

  • 2 hours Lectures and 2 hours exercises / practicals. These are held on Fridays (10:15-12am) and (3:15-5pm))

Check detailed schedule below to know which one takes place when!

Lectures + exercises take place in room CM010.

Practice sesions will be done in the same room and students are expected to bring their own laptop and have related software packages installed.

Software The course encompasses computer-based practicals centred on the use of the MLDemos and ML_toolbox. The first is an open-source software which allows to visualize and test most of the major algorithms seen in class and the second is a matlab equivalent.


Practicals and Mini-Project

The practicals will cover different topics (Kernel Methods, Classification, Regression and Reinforcement Learning). In addition, students will conduct either a mini-project or a literature survey during the course of the semester. The mini-project will entail implementing an algorithm of choice and one or more variants, evaluating the algorithm performance and sensibility to parameter choices. The literature survey will be done on a topic chosen among a list provided in class. The student will read and analyze a choice of recent articles on the topic and provide a written summary. The choice will be announced to the registered students during the semester.

A page dedicated to the mini-project can be found here mini-project webpage

Reports (+code) are due on May 23 2017, @6pm.


Lecture Schedule and Timetable


Date Related Documentation Course Topics Exercises/TP Template Solutions Exercises
Week 1
Feb 24th
(10:15am-12pm)

Concepts
(classification, regression, pattern recognition, etc.)
Methodology
(training/testing sets, leave one out, crossvalidation, ROC, how to measure significance, etc)


Feb 24th
(3:15pm-5pm)



Recap Statistics
(Probability Distribution, Likelihood, E-M, etc.)



Week 2
March 3rd
(10:15am-12pm)


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernels


March 3rd
(3:15pm-5pm)


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernel PCA



Week 3
March 10
(10:15am-12pm)
 


Practical Session 1
Dimensionality Reduction: PCA + Kernel-PCA


 

March 10
(3:15pm-5pm)


Introduction to Mini-Projects
Mini-project + Lit. Review Topics



Week 4
March 17th
(10:15am-12pm)


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernel CCA



March 17th
(3:15pm-5pm)


Spectral methods
(Structure Discovery, Dim. Reduction)

Kernel K-means



Week 5
March 24th
(10:15am-12pm)


Spectral methods
(Structure Discovery, Dim. Reduction)

Spectral Clustering


March 24th
(3:15pm-5pm)


Practical Session 2
Spectral Clustering + Semi-Supervised Clustering



Week 6

March 31th
(10:15am-12pm)


Classification

Support Vector Machine (SVM)


March 31th
(3:15pm-5pm)


Classification

Support Vector Machine and extensions



Week 7
April 7th
(10:15am-12pm)


Classification

Bagging, Boosting + Ransac


April 7th
(3:15pm-5pm)


Practical Session 3
Classification: SVM + Boosting


Week 8
April 14th

+++ Easter Break +++
Friday April 14 -- Sunday April 23



Week 9
April 21st

+++ Easter Break +++
Friday April 14 -- Sunday April 23



Week 10
April 28th
(10:15am-12pm)


Regression

Non-linear regression techniques


 


April 28th
(3:15pm-5pm)



Regression

Non-linear Regression Technniques: exercises



Week 11

May 5th
(10:15am-12pm)


Regression

Other methods: SVR, RVR, LWPR, Gradient Boosting


May 5th
(3:15pm-5pm)


Practical Session 4
Regression: SVR + RVR + GPR



Week 12
May 12th
(10:15am-12pm)


Markov-Processes

---

May 12th
(3:15pm-5pm)
Markov-Processes

---



Week 13
May 19th
(10:15am-12pm)


Mini-Project Practical Session
Work on mini-project / lit. survey


 

May 19th
(3:15pm-5pm)


Mini-Project Practical Session
Work on mini-project / lit. survey


 

Week 14
May 26th
(10:15am-12pm)
 


Student Oral Presentations of Projects / Surveys
Please attend to your colleagues' presentations!



May 26th
(3:15pm-5pm)


Student Oral Presentations of Projects / Surveys
Please attend to your colleagues' presentations!



Week 15
June 2nd
(10:15am-12pm)

Overview of the Class

June 2nd
(3:15pm-5pm)

Q&A Session


TBA

Oral Exam

Scheduled via doodle sign-up.





Online Resources in Machine Learning

Machine Learning.org

On-line repository of papers from NIPS conference

Pascal: Network of Excellence on Pattern Recognition, Statistical and Computational Learning (summer schools and workshops)

ML List: "Archives of the Machine Learning List"

ML Repository: "This is a repository of databases, domain theories and data generators that are used by the machine learning community for the empirical analysis of machine learning algorithms."

MLnet: Machine Learning network online information service. "This site is dedicated to the field of machine learning, knowledge discovery, case-based reasoning, knowledge acquisition, and data mining."

KDnet: "The KDNet (= Knowledge Discovery Network of Excellence) is an open Network of participants from science, industry and the public sector. The major purpose of this international project is to integrate real-life business problems into research discussions and to collaborate in shaping the future of Knowledge Discovery and Data Mining."

Clever Methods for Overfitting


Recommended Textbooks

Selected chapters from some of the book below are made available to the class, see column "related documentation" above.

General Introduction to Machine Learning

Machine Learning: A probabilistic perspective by Kevin P. Murphy, MIT Press

Introduction to machine learning by A. Smola and S.V.N. Vishwanathan, on-line version.

"Bayesian Reasoning and Machine Learning" by D. Barber, Cambridge University Press

 

Kernel Methods: PCA, SVM:

 - "Kernel Methods for Pattern Analysis" by John Shawe-Taylor, Nello Cristianini
   Publisher: Cambridge University Press (June 28, 2004)
  
 - "Pattern Recognition and Machine Learning" by Christopher M. Bishop
   Publisher: Springer; 1 edition (October 1, 2007)

- "Learning with Kernels", B. Scholkopf and A. Smola, MIT Press 2002
  

Statistical Learning Methods:

 - "Pattern Classification" by Richard O. Duda, Peter E. Hart, David G. Stork
   Publisher: Wiley-Interscience; 2 Sub edition (October 2000)
  
 - "Introduction to Statistical Learning Theory" by Olivier Bousquet, Stephane Boucheron, and Gabor Lugosi
   http://www.kyb.mpg.de/publications/pdfs/pdf2819.pdf

- "Information Theory, Inference and Learning Algorithms", David J.C Mackay, Cambridge University Press, 2003.

 

Neural Networks:

- "Spiking Neuron Models” by W. Gerstner and W. M. Kistler
Publisher: Cambridge University Press, Aug. 2002

- "Hebbian Learning and Negative Feedback Networks (Advanced Information and Knowledge Processing) by C. Fyfe.
Publisher: Springer.

- "Independent Component Analysis", A. Hyvarinen, J. Karhunen and E. Oja, Wiley Inter-Sciences. 2001.

- "Self-Organizing Maps", Teuvo Kohonen, Springer Series in Information Sciences, 30, Springer. 2001.

  - "Introduction to Neural Networks: A Comprehensive Foundation" (2nd Edition) by S. Haykins.

 

Reinforcement Learning

- "Reinforcement Learning: An Introduction",. R. Sutton & A. Barto, A Bradford Book. MIT Press, 1998.

- "Reinforcement Learning: A Survey", Leslie Pack Kaelbling & Michael L. Littman and Andrew W. Moore, Journal of Artificial Intelligence Research, Volume 4, 1996., 1996



Last update: 10/02/12