Don't set compiler flags as cache variables

They were added recursively on each cmake configuration, hence
forcing an entire rebuild