Language names in English are uncountable, so this sense takes no a and has no plural: speak Serbian, learn Serbian, in Serbian, a word of Serbian. The capital S stays. It is a dialect of Serbo-Croatian, and it is also the standardized variety of Serbo-Croatian spoken by Serbs, with a historical use where the name covered Serbo-Croatian in general as spoken in Serbia. Those are different ways of describing the same speech.

This one is countable, so it takes a or an and has a plural, Serbians. The capital S stays. Serbian and Serb are treated as the same thing here, and both a native or inhabitant of Serbia and a Serb are listed under this noun. In ordinary use, Serbian tends to point at where someone is from and Serb is the shorter word often used for the people as a group; the line between them is not sharp, so it is fair to describe them as overlapping rather than as two separate things.