As always it depends. Most privacy issues are aimed at better service delivery, customer fit, better understanding of the person.
The border is thin, and it's hard to define. I think it ends up where the data goes beyond a particular site - where the data is sold to others.
For example, if the information about my visit to the site is used only by the owner, let them have it. It is me who goes to his website and looks for products that interest me.