ECE 563 | Fall 2008

ECE 563, Fall 2008
Information Theory

Course Staff

Todd P Coleman (Instructor)
colemant at illinois.edu
Tel: 333-0880
Office hours:
Time: Mondays 4pm-5pm
Place: 118 CSL

Siva Kumar Gorantla (TA)
sgorant2 at uiuc.edu
Office hours:
Time: Tuesdays 4pm-6pm
Place: 125 CSL

General Information

Pre-requisite:
One of Math 464, Math 564, ECE 534 or consent of instructor.

Other listings:
CS 578, Stat 563

Time:
M 8:00-9:50
W 9:00-9:50

Location:
163 Everitt Lab

Credit: 4 hours

Course blurb here

Textbook:
T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley, 2nd Edn.

Homeworks

HW1 (pdf, solutions).
Assigned: 8/24, Due: 9/10

HW2 (pdf, solutions).
Assigned: 9/10, Due: 9/24

HW3 (pdf, solutions).
Assigned: 9/23, Due: 10/15

Exams

Midterm-1: 10/1/2008 7pm-8:30pm, EVRT 269.

Midterm-2: 11/19/2008 7pm-8:30pm, EVRT 269,

Final Exam: TBD

Announcements

Oct 6 - HW3 due date changed to Oct 15th.

Grading
There will be seven (+/-1) homework sets worth 20%, two midterm exams (15% each) worth 30% and a final exam worth 50%.

Topics

Chapter 2: Entropy, Relative Entropy and Mutual Information.
Chapter 3: The Asymptotic Equipartition Property.
Chapter 4: Entropy Rates of a Stochastic Process.
Chapter 5: Lossless data compression (Huffman, Ziv-Lempel, Arithmetic, Shannon-Fano codes): Kraft inequality, Shannon's source coding theorem.
Chapter 7: Channel capacity: jointly typical sequences, Fano's inequality, Shannon's channel coding theorem and its converse.
Chapter 8: Differential Entropy.
Chapter 9: Gaussian Channels.
Chapter 10: Rate-distortion theory: Shannon's source coding theorem relative to a fidelity criterion.
Other topics as time permits.

References

R. G. Gallager. Information Theory and Reliable Communications, Wiley.
D. Mackay. Information Theory, Inference and Learning Algorithms, Cambridge University Press.

Resources

Some useful links