Someone called my phone claiming i called them and wanting to know who i was?

they proceeded to tell me they didn't appreciate me calling them, when i hadn't. then i missed a call from another unknown number, so i called it back and got vm. that person called me wanting to know who i was and why i was calling them. i said they had called me and i missed the call. they said they didn't and didn't appreciate prank phone calls. should i be worry someone has my number and is making crazy calls somehow?