english only
EPFL > STI > IMT > LASA > Teaching > Machine Learning Course
LASA MENU
Home
Research
Publications
Teaching
People
Arriving to LASA
Videos / Photos
Press & News

GitHub Repository
Older Source Codes
The Blog
Intranet

Prospective Students
TEACHING INDEX
Teaching
Student Projects
Former Projects
Robot Contests

MACHINE LEARNING




Overview and objective

The aim of machine learning is to extract knowledge from data. The algorithm may be informed by incorporating prior knowledge of the task at hand. The amount of information varies from fully supervised to unsupervised or semi-supervised learning. This course will present some of the core advanced methods in the field for structure discovery, classification and non-linear regression. This is an advanced class in Machine Learning; hence, students are expected to have some background in the field. The class will be accompanied by practical session on computer, using the mldemos software (http://mldemos.epfl.ch) that encompasses more than 30 state of the art algorithms.



People Lecturer: Aude Billard, Assistant: Basilio Noris
Grading Scheme 50% of the Grade will be based on personal work done during the semester (this entails either doing a computer based mini-project or writing a literature survey, see section Practicals below). The remaining 50% of the grade will be based on a 40 minutes oral exam (20 minutes preparation, 20 minutes oral defense). The oral will examine the student on the material viewed during the course.
Prerequisites Strong knowledge in Linear Algebra, Probability and Statistics


This course is intended as an advanced class in ML for PhD students and hence focuses on advanced topics of ML as well as on critical reading of the state of the art in ML. MSc student who want to take this course are expected to have taken the introductory ML course given at the master during the winter semester.
Lecture Notes
The class is accompanied by lecture notes entitled "Machine Learning Techniques". Hardcopies are available at the librairie polytechnique.
The supplementary material for the lecture notes (examples and demos of the algorithms presented) is available HERE.
Time and Location

Every other week, we alternate between:

  • 2+2 format (with 2 hours Lectures (9am-11am) + 2 hours exercises (11am-1pm))
  • 4-hour format with 4 hours long practical session on computer (9am - 1pm).

Check detailed schedule below to know which one takes place when.

Lectures + exercises take place each friday from 9am through 1pm in room MEB331.

Exceptionally on March 22, the course will take place in room GRC02

Practicals take place in room GRC02.

Software The course encompasses 4 hours of computer-based practicals centred on the use of the MLDemos software (available at http://mldemos.epfl.ch ). This open-source software allows to visualize and test most of the major algorithms seen in class.


Practicals and Mini-Project

The practicals will cover different topics (Kernel Methods, Classification, Regression and Reinforcement Learning). In addition, students will conduct either a mini-project or a literature survey during the course of the semester. The mini-project will entail implementing an algorithm of choice and one or more variants, evaluating the algorithm performance and sensibility to parameter choices. The literature survey will be done on a topic chosen among a list provided in class. The student will read and analyze a choice of recent articles on the topic and provide a written summary.

Link to the Mini-Project page

Results from the computer-based mini-project or survey will be presented in class (10 minutes presentation) and documented in a report (10 pages maximum, 10pt minimum, single column; code from mini-project must be submitted at the same time as the report). Reports + code are due on May 17 2012, 6pm.


Lecture Schedule and Timetable


Date Related Chapter from Lecture Notes Course Slides Exercises/TP Template Solutions Exercises
22 Feb. Chapter 1 + Annexes

Concepts
(classification, regression, density estimation, etc)
Methodology
(training/testing sets, leave one out, crossvalidation, ROC, how to measure significance, etc)

-----


1 March

Sections 2.1-2.2 (must be read before class)

Sections 5.1-5.3.

Spectral methods: Structure Discovery, Dim. Reduction
PCA, Probabilistic PCA, kernel PCA

Exercises

Matlab code for kPCA

Solutions

8 March  

Practicals in room GRC002

Practicals Continued

Assignment, Datasets, Description, Solutions

15 March

Sections 3.1 (must be read before class)

Sections 5.4-5.6.

Spectral methods: Structure Discovery, Dim. Reduction
KCCA, Kernel K-Means,
Supplementary material: Kernel K-means

Exercises

Matlab Code kCCA , Matlab Code ICA , Matlab Code kICA

Solutions(complete)


22 March
Spectral methods: Structure Discovery, Dim. Reduction
Spectral Clustering
Supplementary material: A tutorial on Spectral Clustering, Extensions

Practicals in room GRC002

Assignment, Datasets, Solutions

29 March  

Good Friday (public holiday)

--

--

5 April

Easter Break

--

--

12 Avril
Non-linear Classification

Support Vector Machine, RVM

Supplementary material:on RVM

Solutions

19 Avril

Sections 3.2 must be read before class

Mixtures of Classifiers
Bagging, Boosting and Ransac
Comparison Classifiers

Exercises

Solutions


26 April  

Practicals in room GRC002

Practicals Continued

Assignment Datasets
Solutions

3 May Chapter 4 (must be read before class) Sections 5.8 and 5.9

Non-linear Regression
SVR, GMR, GPR

Exercises

Solutions

Solutions (continued)

10 May  

Practicals in room GRC002

Practicals Continued

Assignment Datasets

17 May Sections 7.1 and 7.3 (must be read before class)

Makov-Processes
Reinforcement Learning

Exercises

  Solutions
Solutions II

24 May  

Makov-Processes
Continuous state-action RL

Overview of class


31 May

Student Presentations
Please attend all to listen to your colleagues presentations

Continued...

--

19 June
Room ME.B1.10

Oral Exam

Continued...

--




Online Resources in Machine Learning

Machine Learning.org

On-line repository of papers from NIPS conference

Pascal: Network of Excellence on Pattern Recognition, Statistical and Computational Learning (summer schools and workshops)

ML List: "Archives of the Machine Learning List"

ML Repository: "This is a repository of databases, domain theories and data generators that are used by the machine learning community for the empirical analysis of machine learning algorithms."

MLnet: Machine Learning network online information service. "This site is dedicated to the field of machine learning, knowledge discovery, case-based reasoning, knowledge acquisition, and data mining."

KDnet: "The KDNet (= Knowledge Discovery Network of Excellence) is an open Network of participants from science, industry and the public sector. The major purpose of this international project is to integrate real-life business problems into research discussions and to collaborate in shaping the future of Knowledge Discovery and Data Mining."

Clever Methods for Overfitting


 

Recommended Textbooks

 

Kernel Methods: PCA, SVM:

 - "Kernel Methods for Pattern Analysis" by John Shawe-Taylor, Nello Cristianini
   Publisher: Cambridge University Press (June 28, 2004)
  
 - "Pattern Recognition and Machine Learning" by Christopher M. Bishop
   Publisher: Springer; 1 edition (October 1, 2007)

- "Learning with Kernels", B. Scholkopf and A. Smola, MIT Press 2002
  

 

Statistical Learning Methods:

 - "Pattern Classification" by Richard O. Duda, Peter E. Hart, David G. Stork
   Publisher: Wiley-Interscience; 2 Sub edition (October 2000)
  
 - "Introduction to Statistical Learning Theory" by Olivier Bousquet, Stephane Boucheron, and Gabor Lugosi
   http://www.kyb.mpg.de/publications/pdfs/pdf2819.pdf

- "Information Theory, Inference and Learning Algorithms", David J.C Mackay, Cambridge University Press, 2003.

 

Neural Networks:

- "Spiking Neuron Models” by W. Gerstner and W. M. Kistler
Publisher: Cambridge University Press, Aug. 2002

- "Hebbian Learning and Negative Feedback Networks (Advanced Information and Knowledge Processing) by C. Fyfe.
Publisher: Springer.

- "Independent Component Analysis", A. Hyvarinen, J. Karhunen and E. Oja, Wiley Inter-Sciences. 2001.

- "Self-Organizing Maps", Teuvo Kohonen, Springer Series in Information Sciences, 30, Springer. 2001.

  - "Introduction to Neural Networks: A Comprehensive Foundation" (2nd Edition) by S. Haykins.

 

Reinforcement Learning

- "Reinforcement Learning: An Introduction",. R. Sutton & A. Barto, A Bradford Book. MIT Press, 1998.

- "Reinforcement Learning: A Survey", Leslie Pack Kaelbling & Michael L. Littman and Andrew W. Moore, Journal of Artificial Intelligence Research, Volume 4, 1996., 1996



Last update: 10/02/12