I am by no means a master of BCNMs, but I've done my fair share
To gain access to a BCNM fight, you need to exchange Beastman's Seals to a vendor in Port Jeuno. Usually, the number of seals to buy the orb corresponds to the level cap of the BCNM... so for a BCNM20, you need to buy an orb for 20 seals.
BCNM fights are generally extremely difficult to win. However, most of them have "ideal" party setups that make it much easier for certain parties with the right job combinations.
There are a few that let a party of beastmasters win easily every time, for example, just by charming the monsters in the BCNM and pitting them against each other.
I discovered, through trial and error, a BCNM 40 that lets a monk, a paladin, and a red mage win easily every time... but a party of beastmasters would die horribly in this one.
There's another BCNM40 full of worms that a party of rangers and black mages can sleepwalk through if they do it carefully enough.
You get the idea.
The most important thing is to do your homework and look at sites like Mysterytour or Allakhazam and learn from the experiences of others. Find out what party combinations work and what party members should and should not do once they get into the BCNM.
BCNMs often fail not because everyone was killed (although that happens a lot too

), but because the time limit ran out. Usually, a 3 person BCNM has a 15 minute time limit and a 6 person has 30 minutes. So it's important to bring party members that are effective at damaging the monster quickly.
The only item of serious value in a BCNM 20 is an Astral Ring. These rings don't drop often when you win, but if one does, it typically fetches just shy of a million gil at the Auction House. You can also get mannequin parts from these.
As for BCNM 30s, if you're looking to earn gil, stick with the ones that drop earrings. Some of them, like the Magician's Earring, can fetch over a million gil. The ones that drop the belts hardly make it worth it.
The BCNM 40s drop Erase and Utsusemi: Ni... both highly prized.