"The dugouts were actually dug out," he said, remembering days spent with other fathers fixing up a small field for their sons on Love Lane in 1958.
February marks the 50th anniversary of the Little League, which started as just four teams put together by a group of dads who wanted a local spot for their kids to play ball.
To have a league, they needed a field. So they started by leveling the ground, growing some grass, and building a backstop - a project that took them about two months.
"It was the most beautiful field you've ever seen," said Olson. "There were trees hanging over the bleachers and you could sit in the shade. We all pitched in - we had a lot of people with a lot of energy."
With a field like that, the league quickly caught on. As Olson remembers, it was just as fun for the parents as the kids.
"I'd rush home from work because the games started at 6. ... It ate up a lot of our social lives. In fact it was our social lives," said Olson, who grew up swinging bats on a sandlot.
In the past half century, the San Ramon Valley Little League has won the national championship three times, competed in a world series against Taiwan, and has earned over 120 flags from district, sectional and regional tournaments.
"The part that's truly amazing is how well the league has done competitively. Statistically, it's almost impossible," said Bill Hooper, who lives in Alamo, coaches and is the president of the Northern California Regional Little League.
The league now hosts 60 teams and more than 700 boys, ages 5-16, in Danville and Alamo. With over 7,400 Little Leagues in the country, winning nationals is an honor most teams never even come close to.
A 50-year anniversary dinner celebration for the league will be held Feb. 1 at Diablo Country Club, and teams from throughout the years will gather for reunions. Local baseball great Tony La Russa, who now manages the St. Louis Cardinals, will speak and be available for autographs.
For Little Leaguers in Alamo and Danville today, baseball isn't about hitting a ball with a bat. It's about feeling - often for the first time - what it's like to have people depend on you, Hooper said.
"It's having to be on time or getting your homework done so you don't let down the rest of the team. That's the part that translates to life," he said.
And being in the team environment helps kids take a swing at some of life's curveballs, said current league president Pat Oswald.
"It's not always easy growing up," Oswald said. "Maybe your parents divorced or you had a bad day at school. But your team is still there and you get to go out and sniff the grass and feel the sun on your face."
Being part of a team teaches children the importance of setting goals and following through with commitments, he said.
"It helps them figure out what direction they want to go. You establish something out of reach and you strive for it," he said.
Since the league restricts the population base to 20,000, the boundaries have changed with the growth of the San Ramon Valley. The original boundaries would now include Danville, Tassajara Valley and Canyon Creek Little Leagues.
In the Danville area, parents still make baseball a priority, Oswald said.
In that way, not much has changed in 50 years.
"There's something in the water out here," he said.
Eventually the Love Lane field was made into a school bus parking lot for San Ramon Valley High School, but the diamond still served 10 years as the league's first stomping grounds.
"It was a memorable time. It became part of our lives - it just took off," Olson said.
Celebrate the League
What: San Ramon Valley Little League Anniversary Dinner
Where: Diablo Country Club
When: 6 p.m., Friday, Feb. 1
Cost: $55 for adults; $30 for children 12 and under
Guest speaker: Tony La Russa will be available for autographs at 5:30 p.m.
Information: Contact Bill Hooper at (510) 446-7244.