In 1919, the U.S. Treasury Annex was constructed on the site where the Freedman's Bank once stood. On December 8, 2015, the U.S. Treasure Department announced that it would commemorate the 150th anniversary of Freedman’s Savings and Trust Company by renaming the Treasury Annex the Freedman’s Bank Building. The name became official on January 7, 2016.

U.S. Treasure Annex