Here is an example of a french house in Normandy, made entirely from marle clay. Obviously there was no source of field stones, or ramdom stone. There are, however, some pieces of cut stone around the front door and the best room window. I took this shot in 2004, but haven't been along that road since, so I don't know what the situation is now. You can see the remains of a lime render on the gable end.
I completely forgot that pics can be pasted into the text.