Yesterday I traveled from Hayes & Harlington to Tooting Broadway avoiding zone 1 and tapped on the pink reader at West Brompton. The Oyster card is linked to a 16-25 Railcard and the entire journey was off peak. However £1.80 was charged instead of £1.00 (single journey price from the price finder) is this a bug or did I do something wrong?

Short answer:

The pink oyster reader did function correctly, in the sense that it has registered your journey as one that avoids zone 1. However, since you used a Southern service in between your journey, you are charged a mixed Transport for London (TfL) + National Rail (NR) fare instead, which is slightly more expensive than a TfL only journey.

The lesson to readers is hence: If your rail journey solely involves routes which is operated under Transport for London (TfL) (Underground, Overground, TfL Rail and DLR), do not take any National Rail routes - you will get charged more. This obviously does not apply if you have a Travelcard, and some other exceptions apply (e.g. journeys on Thameslink core is charged at TfL rate, see last section of the long answer).

Long answer:

What does the alternate fare route actually entail?

For this particular journey (Hayes & Harlington to Tooting Broadway), TfL presented the following way to avoid zone 1 in their single fare finder, which in my opinion is slightly cryptic:

Avoiding Zone 1 via Clapham North/Clapham High Street, Clapham Junction and Shepherd's Bush (or West Brompton and Earl's Court)

This means you have two options in making this journey. Note an OSI below means an Out of Station Interchange - where you have to touch out, and back into another physically separate station for an interchange:

1. Hayes & Harlington (TfL Rail) >
   
   Ealing Broadway (Central line) >
   
   Shepherd's Bush (OSI to London Overground) >
   
   Clapham Junction (London Overground) >
   
   Clapham High Street (OSI to Clapham North for Northern line) >
   
   Tooting Broadway




2. Hayes & Harlington (TfL Rail) >
   
   Ealing Broadway (District line) >
   
   Earl's Court (District Line) >
   
   West Brompton (London Overground) >
   
   Clapham Junction (London Overground) >
   
   Clapham High Street (OSI to Clapham North for Northern line) >
   
   Tooting Broadway

Notice all legs in either journey only involve rails that are under TfL control - TfL rail, London Overground, and lines in London Underground.

Enter National Rail...

Now judging on the journey record, the OP is very likely to have chosen option 2, and touched on the pink reader in West Brompton, though once they arrived Clapham Junction, they took a slight deviation in route.

2. Hayes & Harlington (TfL Rail) >
   
   Ealing Broadway (District line) >
   
   Earl's Court (District Line) >
   
   West Brompton (London Overground) >
   Clapham Junction (Southern) >
   
   Balham NR (OSI to Balham LU for Northern line) >
   
   Tooting Broadway

While it sounded like the OP just took another available train, they have inadvertently straddled the two very different systems, one that is under TfL (and Mayor of London)'s control, and one that TfL has no control at all, including the fares.

As a result, the OP is charged a mixed TfL and NR fare, a fare scale charged when someone take a journey involving both systems. This is always higher than the fare charged when someone takes a journey using TfL's rail system only.

Oyster fare structure is a mess

I will start by quoting Mike Whitaker, a well respected guy in the community who specialises in oyster rail fare. He said on his site (emphasis mine):

Despite setting zonal fares a few years ago, the fare structure for rail journeys is incredibly complex and getting worse. TfL used to have a series of pages on their website which described the various scales in great detail. Unfortunately they decided that maintaining them was too complex, so they removed them.

He did the community a great service by sending a large number of Freedom of Information request to TfL to obtain e.g. the fare information, and created an independent single fare finder which queries TfL's fare database with supplement information.
I encourage you to read his Guide to fare scales, which shows as I write there are actually eight different kinds of fare with Oyster.

Alternatively (for those who wanted to avoid information overload), you can attempt to find the correct single journey fare using this table for adult fares, or this table for railcard holders. One should use the "Off-peak" table, "NR1-T" column, and "Zones 2-5" row to find the correct fare for the OP's journey, which is indeed £1.80.