Fortunately, manufacturers of rack equipment have also realised this and produced some nice modular systems. The one I prefer, because it is flexible, low cost and British, is made by a company called Monacor:
They do a number of pre-punched 19 inch rack mounting panels which are fine if you happen to want the number and type of holes they have available. However, if you want something more flexible they have a 2U frame onto which you can mount a number of different panels:
The basic frame has space for 10 segments which can be filled with pre-punched panels one or two segments wide. Here's a frame with a selection of panels laid on it:
From right to left we have a 2 segment panel punched for Neutrik XLRs into which I have fitted four plugs, then we have the same panel unpopulated, next a 2 segment slotted ventilation panel (very handy for tube mixer designs), then a one segment blank, a one segment panel punched for two TRS jacks and lastly a 2 segment blank which I will use to mount the the IEC mains inlet connector. Monacor do have an IEC pre-punched panel but the mains inlet I am using on this mixer is rather bigger as it includes a fuse, an on/off switch and a mains filter. At least I only have one big hole to make myself. All the other connections needed by the mixer are taken care of by the Monacor pre-punched panels.
The only downside is that there is no means of easily labelling the connectors. Now you can get a different pre-punched panel system from Canford Audio that does include the ability to label the connectors but it is five times more expensive so for the moment I am happy to think of an alternative means of achieving this.
And before anybody asks I have no association with Monacor at all.