The foremost occasion of bigotry encountered by Dr. Smith, at least the one discussed in most biographies, was his rejection by American universities. 

However there is evidence that Dr. Smith encountered racial prejudice elsewhere even amongst the activists he joined for social reform.

… the World’s Temperance Convention: “There was one feature more anomalous than the rejection and gagging of Miss Brown, darker and far more cruel, for it has not the excuse of custom, nor can the Bible be tortured into any justification of it. This was the exclusion of Dr. James McCune Smith, a gentleman graduate of the Edinburgh University (sic), a member of a long-established temperance society, and a regularly appointed delegate. And wherefore? Simply for the reason that nature had bestowed on his complexion a darker, richer tint than upon some of the syncophants who gathered there; it appears to have been simply to pander to a bigoted priesthood and corrupt populace.
– from History of Woman Suffrage, By Elizabeth Cady Stanton, Susan Brownell Anthony, Matilda Joslyn Gage, Ida Husted Harper, pub. 1881

While Dr. Smith championed freedom for African Americans and suppored education for people of color, his own family chose to identify themselves as “white” after his death.