It turns out that Windows protects EFI partition from being deleted. One is able to delete the partion via

• diskpart (Win + R, enter the command, a terminal windows with diskpart running will pop up)
• list disk, will print a list of all disks, e.g.:

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online         3726 GB      0 B        *
  Disk 1    Online          931 GB  1024 KB        *
  Disk 2    Online          238 GB   450 MB
  Disk 3    Online          476 GB      0 B        *
  Disk 5    Online           57 GB    57 GB

• select your relevant disk. in my case, it's Disk five: sel disk 5 (double check your disk numeber!)
• list partition

DISKPART> list partition

  Partition ###  Type              Size     Offset
  -------------  ----------------  -------  -------
  Partition 1    System            3968 KB  1988 MB

DISKPART> sel partition 1

And finally, deleted the partion via:

DISKPART> delete partition override
DiskPart successfully deleted the selected partition.

From the source, they also set an ID, yet that part both did not work nor was it necessary for me.

Now the entire USB stick was deleted:

Solution: Use Ubuntu disk utility. It managed to delete one partition and reformat the usb stick.

The article How to Delete EFI System Partition in Windows recommends using diskpart for deleting the partition, as follows.

The first step is to change the partition id, in effect converting it from EFI to data:

list disk          (displays all disks)
sel disk N         (select the EFI disk by its number N)
list partition     (displays all partitions including EFI)
sel partition N    (select the EFI partition by its number N)
SET ID=ebd0a0a2-b9e5-4433-87c0-68b6b72699c7

Once this is done, to delete the EFI partition with diskpart:

delete partition override

If successful, you will get the message "DiskPart successfully deleted the selected partition". Use the exit command to close diskpart.