How did castles decline in popularity?

With the introduction of gunpowder in the 15th century, the once mighty castles fell with relative ease to invaders. However, opposed to popular belief, it wasn't really gunpowder that finished the Castles Age. Most castles could be strengthened to stop even cannonballs.
Many castles became prisons as it wasn't only difficult to get in, but also to get out. The Tower of London and the Bastille are notorious for hosting primarily political prisoners.