Sebastian was originally constructed and named by the Portuguese c.1520-26. However, its first appearance on a map was in the context of Di Castaldi's Venetian map of 1564. In putting up the fort, the Portuguese hoped to stop English ships from interfering in their trade in the Shama area.

According to a Portuguese chart of 1630, the Portuguese fort had a bastion, two single-storeyed buildings with pitched roofs and a two-storeyed tower. At the time the Dutch took over the fort it was in a state of ruin. While retaining its name, they carried out repairs in 1640-42 and added substantially to the previous structural form.

Built as a Dutch lodge in 1526. Portuguese fort built in 1590. Abandoned in 1600. Restored and altered by the Dutch c.1638, enlarged in 1640-2. Attacked by the English under Captain Robert Holme. Temporally in English hands, in 1664-65. The Dutch struck back under De Ruyter, re-occupied it the same year and rebuilt it in 1666. Abandoned it before 1870. Ceded to Britain in 1872. Restored in 1954-57.