Two separate developments involving improved relations between Sunni and Shia Muslims and women's sporting rights demonstrate major shifts in how rivalry for the leadership of the Muslim world and competition to define Islam in the 21st century are playing out in a world where the Middle Eastern states can no longer depend on the United States coming to their defence.

The developments fit into a regional effort by conservative, status-quo states like Saudi Arabia, the United Arab Emirates (UAE), and Egypt and proponents of different forms of political Islam (Iran, Turkey, and Qatar) to manage—rather than resolve—their differences in a bid to ensure that they do not spin out of control. The efforts have had the greatest success with the lifting of a 3.5-year Saudi-UAE-Egyptian-led diplomatic and economic boycott of Qatar in January this year.

The reconciliation moves also signal the pressure on Middle Eastern players in what amounts to a battle for the soul of Islam to change perceptions of the region as being wracked by civil wars, sectarian tensions, extremism, jihadism, and autocracy. Altering that perception is key to the successful implementation of plans to diversify oil and gas export dependent economies in the Gulf, develop resource-poor countries in the region, tackle an economic crisis in Turkey, and enable Iran to cope with the crippling US sanctions.

Finally, these developments are also the harbinger of the next phase in the competition for religious soft power and leadership of the Muslim world. In a break with the past decade, lofty declarations extolling Islam's embrace of tolerance, pluralism, and respect for others' rights that are not followed up by deeds no longer cut ice. Similarly, the proponents of socially conservative expressions of political Islam need to be seen as adopting degrees of moderation that so far have been the preserve of their rivals, who prefer the geopolitical status-quo ante.

That next phase of the battle is being shaped not only by doubts among the US allies in the Middle East about the United States' reliability as a security guarantor—reinforced by America's withdrawal from Afghanistan—but it is also being informed by a realisation that neither China nor Russia can (or will) attempt to replace the US defence umbrella in the Gulf.

The battles' shifting playing field is further being determined by setbacks suffered by political Islam, starting with the 2013 military coup that toppled Mohamed Morsi, a Muslim Brother and Egypt's first and only democratically elected president, and brutally decimated the Muslim Brotherhood. More recently, political Islamists suffered a stunning electoral defeat in Morocco and witnessed the autocratic takeover of power in Tunisia by President Kais Saied.

A newly published survey of Tunisian public opinion showed 45 percent of those polled blaming Rachid Ghannouchi, the leader of the Islamist Ennahada party, for the country's crisis, and 66 percent saying they had no confidence in the party.

The Middle East's rivalries and shifting sands lend added significance to a planned visit in the coming weeks to Najaf, an Iraqi citadel of Shia Muslim learning and home to 91-year-old Shia religious authority, Grand Ayatollah Ali al-Sistani, by Ahmed el-Tayeb, the grand imam of al-Azhar, Sunni Islam's foremost historic educational institution. The visit takes place against the backdrop of Iraqi-mediated talks between Saudi Arabia and Iran, the two major centres of Islam's two main strands, that are aimed at dialling down tensions between them which reverberate throughout the Muslim world. The talks are likely to help the two regional powers manage rather than resolve their differences.

The rivalry was long marked by the Saudi-inspired, religiously-cloaked anti-Shia rhetoric and violence in a limited number of cases, and the Iranian concerns about the country's Sunni minority and its opting for a strategy centred on Shia Muslim proxies in third countries and support for the regime of Syrian President Bashar al-Assad.

Implicit in Saudi and Iranian sectarianism was the perception of the Shia minorities in Saudi Arabia and other Sunni majority countries, and Sunnis in Iran and Iraq after the 2003 toppling of Saddam Hussein, as fifth wheels of the other.

Imam el-Tayeb's visit, a signal of improvement in long-strained Egyptian-Iraqi relations—as well as a possible later meeting between the Sunni cleric, a Shia cleric other than Ayatollah al-Sistani as he is too old and fragile to travel, and Pope Francis—are intended to put sectarianism on the backburner. Ayatollah al-Sistani met with the Pope during his visit to Iraq in March.

The visit takes on added significance in the wake of this week's suicide bombing of a Hazara Shia mosque in the northern Afghan city of Kunduz that killed at least 50 people and wounded 100 others. The South Asian affiliate of the Islamic State, Islamic State-Khorasan, claimed responsibility for the attack—the worst since the Taliban came to power in August. It was likely designed to fuel tension between the Sunni Muslim group and the Hazaras, who account for 20 percent of the Afghan population.

Imam el-Tayeb's travel to Najaf is likely to be followed by a visit by Mohamed al-Issa, secretary-general of the Saudi-dominated Muslim World League (MWL). The MWL was long a prime vehicle for the propagation of anti-Shia Saudi ultra-conservatism. Since coming to office, Crown Prince Mohammed bin Salman has recast the MWL as a tool to project his vaguely defined notion of a state-controlled "moderate" Islam that is tolerant and pluralistic.

In a similar vein, hard-line Iranian President Ebrahim Raisi took many by surprise by allowing women into Tehran's Azadi Stadium to attend this month's World Cup qualifier between Iran and South Korea. Iran is the only country to ban women from attending men's sporting events. It was unclear whether the move was a one-off measure or signalled a loosening or lifting of the ban. Raisi was believed to see it as a way to rally domestic support and improve the Islamic republic's image as much in China and Russia as in the West. No doubt, Raisi will have noted that China and Russia have joined the United States, Europe, and others in pressuring the Taliban in Afghanistan to recognise women's rights.

To be sure, women in Iran enjoy education rights and populate universities. They can occupy senior positions in business and government even if Iran remains a patriarchal society. However, the ban on women in stadiums, coupled with the chador—the head-to-foot covering of women—has come to dominate the perception of Iran's gender policies. Allowing women to attend the World Cup qualifier suggests a degree of flexibility on President Raisi's part. During his presidential campaign, he argued that granting women access to stadiums would not solve their problems. It also demonstrates that the government, with hardliners in control of all branches, can shave off sharp edges of its Islamic rule far easier than reformists like Raisi's predecessor, Hassan Rouhani, were able to do.

The question is whether that is President Raisi's intention. He may be testing the waters with this month's football match; only time will tell.

It may be too big a leap in the immediate future but, like Imam el-Tayeb's visit to Najaf, it indicates that the dialling down of regional tensions puts a greater premium on soft power, which in turn builds up pressure for less harsh expressions of religion.

Dr James M Dorsey is an award-winning journalist and scholar, and a senior fellow at the Middle East Institute in the National University of Singapore.