What file do I need to edited to customize the filter brick?

The fields are rendered in a same way than a regular field. See the /models/fields/... .PHP
They are configured in regular form See the /models/forms/... .XML
The filters layouting is written in the template.


Hope it helps.
I took time to answer because I had to check and remember the actual given way.
Joomla legacy / Joomla 3.4 / FOF / Cook actual proposals ...
There have been many changes in the past because Joomla was lacking.

Now the 'pure' version will furnish the 'almost' final proper way. And also the Ajax fields will be unified such as a native field.
Filters has been a little pain recently for rewrite.

I will explain better in the pure version.