When is "Y" used as a vowel?

I was always taught that the vowels in the English language are A,E,I,O,U and sometimes "Y". Exactly when is it safe to consider "Y" a vowel ???