The world as we perceive it is structured into objects, actions and places. In this talk my aim is to explain why these categories are cognitively primary. From an empiricist and evolutionary standpoint, it is argued that the reduction of the complexity of sensory signals is based on the brain’s capacity to identify various types of invariances that are evolutionarily relevant for the activities of the organism. It is shown that the invariances that determine places, objects and actions have their separate characteristics and that they are, by and large, independent of each other. This separation is supported by what is known about the neural mechanisms. The second aim is to argue that the category of numbers can be analysed along similar lines. Here fungibility will form a central invariance.