What's the Name of That Book??? discussion

I started reading a book in the 7th grade (2008-2009), and at the time I just didn't get it, but it seemed interesting enough for me to want another go. The thing is, I can't remember much at all.
The story, as far as I can tell, focused on a boy who grew up on a small island with a fort of sorts, filled with other boys and, I think, their master. The boy might not have a name, but I could be jumbling this up with details from "Crispin."
They all have jobs, and they get punished for stuff. There's a whipping scene at one point, I believe, and possibly sort of ringed bare-knuckle boxing matches (like in Robert Downey Jr.'s Sherlock Holmes). He might become almost like a son to the master.
This is where it gets real foggy. That was all exposition. I'm fairly certain the island gets raided by pirates and stuff gets set on fire. I also think he might steal from a buried chest that the master keeps jewels and gold in. He escapes into the sea on a boat with another boy. Again, all of that is very foggy for me.
That's where I gave up, and the rest is just from thumbing around. He meets an old lady that feeds him and is kind, he either falls in with or has a run in with a gang of bandits, and that's about it.
It's an adventure novel, I believe, and it was on the shelves of my 7th grade homeroom, so it couldn't have been too far outside of that reading level. I remember it being a little slow and dry (although I had also read the entire Lord of the Rings series by the 4th grade, so my reading level is a little whacky).
If anyone can help me figure this out, that would be amazing. This is driving me crazy.

The Coral Island by R.M. Ballantyne?

Juels wrote: "The Coral Island by R.M. Ballantyne?"
Unless the Wikipedia article fails to include the exposition I described this doesn't seem right. Thanks, though!

This sounds like The Wings of a Falcon to me.

The description from google books is closer to what you remember: Fourteen-year-old Oriel and his friend Griff flee the slavery of Damall's Island and seek a new life on the mainland, where they face raiding Wolfers, rival armies, and other dangers.

I haven't read this one but I'll throw it out there anyway... The Hilltop Boys on Lost Island by Cyril Burleigh?

Adele wrote: "This sounds like The Wings of a Falcon to me.

The description from google books is closer to what you remember: Fourteen-year-old Oriel and his..."

YES this seems extremely familiar, both from the description and the cover. THANK YOU!

Happy to help. :-)