This is the people sense, so it counts: a Xhosa, two Xhosas. The plural Xhosas is the form given in the source, and you will also meet the Xhosa used for the whole group, as in the Xhosa are one of the largest groups in South Africa. The word is a proper noun, so it always keeps its capital letter, wherever it sits in the sentence. Bantu here names a large family of peoples and languages across southern and central Africa, so it places the Xhosa inside a much bigger group.

This is the language sense, and like other language names it is uncountable: no a in front of it, no s on the end, and usually no the either. You speak Xhosa, you learn Xhosa, a sign is written in Xhosa. A click is a sound made by pulling the tongue sharply away from the roof or side of the mouth, and the source picks it out as the feature this language is known for. Saying it is related to Zulu tells you the two share a family, in the way Taiwanese and Hakka sit in one larger family; it does not mean speakers automatically understand each other.