You probably want less fidelity, not more. Fidelity controls how close it tries to match the 3d-structure . Since this building has a lot of texture on the surface, its going to favor bricks or some rough surface to match the existing 3d-structure.
Less fidelity gives the AI more freedom to just paint a flat white surface that doesnt exactly match the existing structure.
this is first try with 0.65 fidelity masking only the side:
