To detect encolsed spaces you'd need to write a flood-fill style detection, using Djikstra for example. Even then if tile placement is completely freeform you'd face the problem of intelligently choosing a spot where to start the detection. If player places wall tiles in some random shape, it would be difficult to tell where is the inside and where is the outside, to start checking for enclosed space. Giving a rectangle draw tool as suggested above to shape buildings is a much more straightforward option, and you can always allow expanding them with the same tool by overlapping rectangles. Also, unless the size of the building affects their functionality, you'd be better off avoiding this headache by giving the players a list of buildings with predefined footprints.
Edit - have you ever played Dward Fortress? I just recalled it has a different approach to defining buildings / workspaces / areas. You can build walls and dig caves and set doors, but if you don't go further than that they're just partitioned spaces with no special functionality. When you want to define a use for an area, you go to another menu and tell the game "I want to place A THING at A SPOT!" and it gives you a selector you point at wherever. You can expand the selection area (initially just a single tile) vertically and horizontally, and it intelligently detects walls and doors and won't expand beyond them. When you ok the selection, those tiles will be marked as whatever thing you wanted. You can also use the tool in inverse, to chip away pieces from an existing definition. By separating building and definition phases the game doesn't need to work with any complex mechanisms.