The human mind is built to notice tiny inconsistencies. I found this interesting article about how web designers have to adjust UI elements, often for the same reason we kern type – it’s just a tad bit “off”.
Yes!! I usually start buy aligning everything mathematically, but then going back for an optical alignment/kern once I’ve got general placement. I work in box packaging design, and when people want alignment between box panels sometimes you have to massage it optically.