Coins are stored in so-called exchange wallets. Here you can trade with them, but also just leave them like you would with a bank account. Since there are different wallets, you also have the option of transferring your coins from one wallet to another. Only you have access to your wallet with an electronic key, the private key.
Most wallets are online, but there are also some hardware wallets. You can think of it as a kind of USB stick. These have the advantage that the coins on them are protected from any hack attacks.