Frankly, this is a question I hadn’t ever given much thought about it – until I ran across an article that raised the question, and then answered it. Now, making a quick jump to say that the phrase likely comes from “of clock” or “of the clock” isn’t too much of a stretch. But why would that even be of a concern? Where else would you be telling the time from?
And it’s that question that sheds the light. The phrase “o’ clock” is actually a holdover from when clocks weren’t all that prevalent, and all manner of methods to tell time were used. So, in order to distinguish that you were telling time by a mechanical clock of some sort, rather than, say, a sundial, you’d say the time was “of the clock”. This, then, got shortened down to o’ clock, and even had a brief period where it was even shorter. For a while in the 16th and 17th centuries, the “o” was omitted, meaning you’d say the time was “6 clock”, for instance. Then again, we can easily omit the word clock these days, and people know what you’re saying.
If you’d like a more in-depth look at this topic, as well as why the “o” got added back in to the phrase, check out this article from Today I Found Out. This is actually a really great resource, and you can get all sorts of information emailed to you daily (not necessarily of the horological variety). So, if you’re the curious sort, you may find this to be an interesting resource.