To remove fields from your import mapping, go to "Setup > Setup Project Fields". You will then see the option to remove a field, as shown in the image below.
It is important to know that if the field is still used somewhere in your import, it will be re-added when importing. The only way you can use this feature is when the field does not exist in your import (anymore) and is not used anywhere in the rules, exports, APIs or Ads. If the field is still being used somewhere, you will see a warning message appearing showing you where the field is still being used.
By clicking on each line in the warning you can clean the rules, exports, APIs, etc. from the used field and after that you can remove the field from the import mapping.