If you don't establish your own presence, that doesn't shield you from being on the internet. It only means you have no idea what is being said about you. People have always talked to each other about brands, but now we can monitor much of that conversation. Jacob Morgan has a great post titled Do companies have control over their brands that you should read.
Get criticized, respond correctly to your critics, and use that exchange to make a better business. It's another tool in your toolbox, you may as well use it to build something great. I'm not telling everyone to start a Twitter account, or a Facebook fan page, those are not for everyone. But there are many other directories, user review sites, and other ways to connect with your customers. Google yourself, see what happens.