I want my players to be able to kill a Mimic and get the loot from the Mimic's corpse.

The Mimic has an acidic bite, and is a creature, not actually a container.

Can a Mimic store items?

Absolutely

Shapechanger. The mimic can use its action to polymorph into an object or back into its true, amorphous form.

A chest has an inside and an outside. So a mimic polymorphed into a chest has an inside and an outside.

Mimics are also not-unintelligent and a mimic could see great utility in being an open chest and then using its Pseudopod to fill itself up to overflowing with nice, tempting treasure.

Now, we know "It reverts to its true form if it dies" but “amorphous” means having no fixed defined form. Maybe each mimic has its own “true, amorphous form”. Or maybe they look like this:

If so, that juicy treasure that was inside the chest would probably be lolling about inside that remarkably large mouth.

It's fairly common in published adventures for e.g. a roper to have loot in its gizzard. It occurs multiple times in Tales from the Yawning Portal.

The monster's manual even suggests treasure winds up there. From page 261.

A roper can digest anything it eats with the exception of platinum, gemstones, and magic items, which can sometimes be retrieved from the creature's gizzard after death.

There's no reason your mimic couldn't have something similar, though I suppose it's not really explicitly spelled out as a common mimic characteristic.

Probably not

Because the mimic has the always on property of Adhesive when in object form:

The mimic adheres to anything that touches it.

It seems unlikely that someone was able to get something inside of it. Once it moves, it's no longer the object and is instead it's true form. And I'd saying that it opening is it moving (because it's moving.)

However, should whomever is using it want to store something inside it (cruel!), I'm not sure that would be totally against the rules.

Storing items inside creatures

I guess technically it can 'contain' the item - but the players are going to have to 'dig' for it.

The mimic is only an object when it's using it's shapechanger ability (my emphasis):

The mimic can use its action to polymorph into an object or back into its true, amorphous form. Its statistics are the same in each form. Any equipment it is wearing or carrying isn't transformed. It reverts to its true form if it dies.

If they've killed it, then it's no longer acting as the 'container', but I don't see why it can't still have something inside it (heck, any monster could have items implanted, I guess.)

As a container, no.

When the mimic dies, it reverts into it's true shape, which is not a container which would normally contain loot. It could have eaten something like an enchanted ring by mistake or something while feasting on the poor adventurer before you, but otherwise, I think it would be on the outside of the Mimic due to its Adhesive property. Now, other monsters do have the ability to have loot inside them, as @kyle sexton has said, like the Roper, which is actually stated in the MM:

A roper can digest anything it eats with the exception of platinum, gemstones, and magic items, which can sometimes be retrieved from the creature’s gizzard after death. A roper’s digestive juices are also valuable, fetching a high price from alchemists who use them as a solvent.

Then you have the Purple Worm which eats through the earth, and can have gems and the like inside its stomach, which is also referenced in the MM:

When a purple worm burrows through the ground, it consumes earth and rock, which it breaks down and constantly excretes. Precious metals and gems can thus be found within the bodies of purple worms, which are targeted by particularly brave and foolhardy treasure hunters.

So... For the mimic, I would think not, but as DM, you could rule it so. Other monsters consume things that they cannot digest and you can loot them for it, so why not the mimic?