Adhesion hysteresis was investigated with the energy dissipation in the contact experiments between a spherical glass lens and a polydimethylsiloxane (PDMS) block. The experiments were conducted under step-by-step loading–unloading for the spontaneous energy dissipation. The force, contact radius, and displacement were measured simultaneously and the elasticity of the PDMS was confirmed. The work of adhesion was estimated in the loading process of the strain energy release rate. The total energy dissipation has been observed to be linearly proportional to the contact radius in the unloading process. The approximately constant gradient of the energy dissipation for each unloading process has been found. The result would provide how the dissipation is induced during the unloading as some interfacial phenomena. The fact has been discussed with some interfacial phenomena, e.g., the adsorbates on the surface, for the mechanism of adhesion hysteresis.