Hi Anurag,
Yes, we need to list all the fields in package.xml. As per the documentation the CustomField metadata type does not support wildcards sadly. You can access Custom Fields on Standard Objects by using the CustomObject metadata type as described hereagainst specific Standard Objects. Then parse the .object file to build your destructiveChanges.xml file. Finally you can also use the sf:listMetadata task to download a list of all custom fields and then filter locally for Standard objects.
<sf:listMetadata
username=”${sf.username}”
password=”${sf.password}”
metadataType=”CustomField”/>