If My Baby Has Milk Allergy Will Consuming Yoghurt Also Irritate Him ?

I breast feed my baby. My baby seems to have milk allergy. Question: Do I also need to stop taking yoghurt ? Is it possible for my baby to be affected by my consumption of yoghurt ?