This word names a people, not a country. Malays are the biggest group in Malaysia, which is why the learner dictionary explains it that way. But the group reaches past that one country: there are Malays in Singapore, in Brunei, on Sumatra and other Indonesian islands, and in southern Thailand. So Malay and Malaysian are not the same word. Malaysian tells you which passport someone holds. Malay tells you which people they belong to. A Malaysian family can be Chinese or Indian and not Malay at all. The noun is countable: a Malay, two Malays. It keeps a capital M.

The adjective has a wide reach. It can point at the people, as in a Malay family. It can point at the country or region, which is how the learner dictionary puts it: from or connected with Malaysia. And it can point at the language, as in a Malay word or a Malay dictionary. It goes in front of a noun and it has no comparative: you do not say more Malay or most Malay. It always keeps a capital M, like English or Japanese.

This is the name of a language, so it is uncountable and takes a capital M. You say speak Malay, in Malay, learn Malay, with no a and no the. The word has a wide use and a narrow use, and both are real. Used widely, Malay covers the whole language, including the standard form that Indonesia calls Indonesian. Used narrowly, Malay means the language without that Indonesian standard. Which one a writer means usually shows from what they are talking about. One thing learners often notice first: Malay makes a plural by saying the word twice, so buku is book and buku-buku is books.