Beastliness is the noun for being beastly: the quality of being very unpleasant, or an act that feels cruel, nasty, or ugly in spirit. You can talk about the beastliness of a remark, of the weather, of the way someone treated a weaker person. It has an old, slightly storybook sound. Grandparents and British novels use it more than teenagers in a chat. Though actually, it sits on two feelings at once. One is 'like a wild animal' — rough, without kindness. The other, and this is the sense you will meet in ordinary English, is simply 'very nasty,' with the animal picture faded. It is not a scientific word for animals. It is a human complaint, sometimes half-joking ('the beastliness of Monday mornings'), sometimes quite serious when someone has been mean.