Do I really have to connect my calendar and contacts?

Yes. If you do not sync your calendar and contacts with Zoom.ai, you will not be able to use the service.