Originally, the lineup change in the Avengers (Thor, Iron Man, Giant-Man and the Wasp leaving) was to concentrate on characters who didn't have their own books and avoid continuity issues.  But like almost everything else in comics, somebody took the idea and ran with it and now everyone's an Avenger, and all the big names are shoehorned in.

The Avengers to me are still centered around Henry Pym, the Wasp, Captain America, Thor, Iron Man, the Scarlet Witch, Hawkeye, the Vision and a few others who come and go (like Quicksilver, Black Panther, Hercules, Wonder Man, Carol Danvers, etc.).

Michael, thanks for the background on the Agents.

The Wasp was just Ant-Man’s sidekick in those books. It’s weird reading them with a modern interpretation of the Wasp in mind. 

The Legion of Super-Heroes: the original three legionnaires were Cosmic Boy, Lightning Lad, and Saturn Girl. They had one or two adventures together before the next members were added - Triplicate Girl and Phantom Girl. By the time of "Superboy and the Legion of Super-Heroes", there were indeed, as you noted, several members - but it started with those three in R.J. Brande's office.

The X-Men's purpose was, as I inferred, to find mutants, help train them to use and control their powers, and then help them fit into society with homo sapiens. Professor X often said how he dreamed of a world where mutants and humans lived and worked together in harmony. Join the X-Men, become a milkman, go into politics - as long as it fit into a good society, I think that was all he wanted.

The Avengers started as a strip putting together all Marvel's then-published super heroes except for the Fantastic Four (in their own team obviously) and Dr. Strange and Spider-Man. I don't know why they were left out, save that Stan may have realized the Dr. Strange just wouldn't fit into a team book, and Spider-Man might have been so strong that putting him in TWO books might have diluted his sales. But that's only a guess.

The Justice League of America was much the same, although working on the previous model of the Justice Society. Pretty much every super hero with a strip was a member of the JLA, save for Green Arrow - we do not know why - and Superboy, obviously. Even the stories kinda matched up... Superman and Batman didn't take part very much for the first ten issues or so.

As the DC heroes increased, the JLA was bound to expand, although after 31 issues, they stayed level for a while; with 11 heroes to draw from, the mix-and-match made it easier to writer stories. The Avengers, on the other hand, really did go to the "no strip of their own" feature heroes, save for Cap - and even HE got de-emphasized some after a bit.

I think that the overloaded teams really are a bit much - but the writer who feels he has to use every single hero also is a big part of that problem.

"Final Crisis: Legion of Three Worlds" seemed to me to be a pretty special even for the crossover. And even at that, Earth-1's Legion was still only half represented. I also feel that the final sequence in Limbo was pretty heavy... but it only was a two page spread, and meself being a LONG time Legion fan, I enjoyed the hell out of it. Even if some of those characters were... a little varied in abilities.*

But that was obviously a one time special event, in the middle of yet another of a series of crises. The Legion has been mistreated over the past few years... I think it was nice to give us LSH fans a little treat.

Still, I wouldn't want it every three months. And some "events" seem to demand such. None for me, thanks.

*Varied in abilities... in that spread of Legionnaires were Earth-1's Supergirl, Comet, Krypto, and Beppo. Three REAL Earth-1 Kryptonians and an Earth-1 pseudo Kryptonian. Seriously... what COULDN'T that group do? We're talking time travel, moving planets, truly super intelligences (almost Brainiac 5 status.... say, level 10 vs. Querl's level 12.) To me, it seemed a little anti-climactic to have those characters available.)

In the team's second chronological appearance, they identified themselves to Supergirl as the children of the three heroes who had met Superboy. Since this accommodation was unnecessary for a team of time-travelers, it was quickly forgotten and never spoken of again. Reprints of the story have that panel relettered to erase any confusion. 

Had that been allowed to stand, however, it would have explained any questions as to who those members sitting in the shadows were in the first story. They were members of the parent's group. 

Of course, it would have been a continuity nightmare to constantly keep track of which members of which team met Superboy and which met Supergirl... Later, when the Legion of Super-Villains showed up, in costumes which exactly matched those of the parents, but with slightly different powers, no doubt a story would have been written which explained that members of the first group had flown through an evil space cloud, reversing their essential good natures, altering their powers and personalities...

Fortunately, none of that was necessary since the idea of two generations of Legionnaires was done away with pretty quickly. Unfortunately, those other members still exist in that first story, so we're left with the idea that the LSH's first appearance occurred later than some of their other appearances, and they didn't go back in time to visit Superboy and Supergirl until after they'd gained some of their other members.

Which still leaves us with the question of why they switched costumes then for that one 1958 story and switched back again to go meet Supergirl... Which is just the sort of silliness that results when one tries to hammer together a continuous, single story out of multiple incidents which were never intended to fit together so closely. 

So, the official in-story origin of the team in Adventure #350 places Garth, Imra, and Rokk as the founding members at the start of the team, yes, but the actual flow of events as published from 1958 forward has always portrayed them as members of a larger organization. Up until that origin story, the Legion was never shown to be comprised of only three members. It was always a somewhat larger group, even if the exact membership was nebulous at times. 

It was at some point in the late 90s that I realized they had been handily surpassed in sheer quantity by the Justice League, the Avengers and the X-Men, with the Teen Titans coming up close behind.