Right click on the album / media name in the left panel where you see the exclamation point. Then select delete from the pop up menu.