Many of these older stories would need some serious editing in today's days. The whole first chapter was too long, added nothing to the story, and it was incredibly boring. This coming from someone who is used to read many classics.
The story is too slow paced, nothing ever happens much, and the little that happens takes a whole chapter. And nothing happens in other chapters rather than some ruminations.
The life of close minded people in the age of Puritanism. They were too eccentric for Europe, so they were glad to move to the New World. In there they came up with their even stricter rules and close-minded ideas. And they judged people very easily, more so if you were a woman.
We follow Hester from the moment she comes out with the Scarlet Letter on her chest as part of her punishment for having sex with another man. A scandal for Puritans. Nothing much nowadays, just people enjoying themselves and their bodies.
Hester, being Puritanical, punishes herself, lives aside od the general populace as muchas possible, and that self-punishment ends up softnening people around her. She has a wild child, fruit of that forbidden relation, whom Hester also takes it as both punishment and the most precious thing she has. 
And time reveals who was the man that should have shared half his shame and guilt publicly, but does it in secret.