I used coiled roll pins to connect the shift arms to the shafts. Working good so far, and are nice and tight. They come in various sizes, and can be bought in small quantities (2) from most industrial fastener suppliers.
https://www.mcmaster.com/roll-pins