Yes! Just create a conference for the total number of people you want.  Then, include those members without e-mail by giving them the date and time of the conference, the phone number to dial and the Participant Access Code.  They will be able to join in without the e-mail invitation and RSVP features.