Depends on your definition of "house".
Technically it is possible to build a type of "shelter"
But to build a real house, even a small shed, it is necessary to use at the very least basic math. Even if you were building and planning on the go, it would require some math. You need math to calculate how large of materials you would need, how much money, the length and width of the house, how much supplies you need etc.
