Just like with all TCP/IP traffic, it is easy for a snoopy person to sniff unencrypted VoIP packets and record your conversations. And don't think they won't.
Remember the early days of cell phones, when people used ordinary police scanners to eavesdrop? Newt Gingrich, Nicole Kidman and Tom Cruise, Prince Charles, and hosts of other celebrities learned the hard way the value of using cell phones with strong encryption.
It's not as easy to snoop wired IP traffic because you need physical access to the wires, but it's not that hard, either. Anyone on your network, anyone on other networks that you contact—and all points in between, including service providers—all have the opportunity to do an awful lot of juicy snooping. Throw in some poorly secured wireless access points, rogue wireless access points, or wireless VoIP endpoints, and you have a real security risk.
While spying on other people's communications is mostly illegal, that's small comfort if it happens to you. (more)(one good solution)