Your thinking is in the right direction. Indeed, if you are given a book under contract, then you have to abide by the contract. But let's say you decided to break the agreement and copied the book on the Internet or gave it to your friends. All those people made no contract with the author and thus they cannot be and should not be bound by the contract. What copyright does is bind third parties, which never agreed to the contract.
There are other cases. Public broadcasting is one. You turn on your TV and see Harry Potter. You were not presented with any contract. Why should you be bound by any conditions? If the author of Harry Potter wanted to distribute her book only under contract obligations, she should not have let it broadcast on TV.
The other case is the one I mentioned - someone breaking a contract.
There are other options, like accidents. You leave a book on a bench. Unless the book itself contains a full blown contract in it, there is no way for Joe who finds the book to know. In fact, even if the contract is in the book, Joe did not sign it.
Additionally, some things are just out there. You invent something. The other person sees it. He has no contract with you. Why should he restrict actions with his own body and his own property? What legitimate reason is there to not act on a new knowledge you have?
Finally, if each time you enter a store and decide to buy a DVD, you were explicitly told that you will not simply be able to share copies, but that you should agree to not use the knowledge you gain from the DVD and act as if you have no such knowledge, I am not sure most people would agree.
You watch Harry Potter. And, according to the contract a typical copyright proponent would want you to sign, you hen have no right to act on the knowledge you have acquired, namely, the plot, ideas, characters. All of this is knowledge you now have in your brain. Copyright wants you to pretend you only have this information in read-only format.