The menu has always been left opened during a customize operation, it's intentional, so you won't miss all the various tooltip notifications that wouldn't appear if it was closed.
We might hide it, but it's something we need to add, it was never there before.